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Preface 



Science consists in using information about the world for the purpose of predict- 
ing, explaining, understanding, and/or controlling phenomena of interest. The 
basic difficulty is that the available information is usually insufficient to attain 
any of those goals with certainty. 

In these lectures we will be concerned with the problem of inductive in- 
ference, that is, the problem of reasoning under conditions of incomplete in- 
formation. Is there a general method for handling uncertainty? Or, at least, 
are there rules that could in principle be followed by an ideally rational agent 
when discussing scientific matters? What makes one statement more plausible 
than another? How much more plausible? And then, when new information 
is acquired how does it change its mind? Or, to put it differently, are there 
rules for learning? Are there rules for processing information that are objective 
and consistent? Are they unique? And, come to think of it, what, after all, 
is information? It is clear that data "contains" or "conveys" information, but 
what does this precisely mean? Can information be conveyed in other ways? Is 
information some sort of physical fluid that can be contained or transported? 
Is information physical! Can we measure amounts of information? Do we need 
to? 

Our goal is to develop the main tools for inductive inference - probability 
and entropy - and to illustrate their use in physics. To be specific we will 
concentrate on examples borrowed from the foundations of classical statistical 
physics, but this is not meant to reflect a limitation of these inductive methods, 
which, as far as we can tell at present arc of universal applicability. It is just 
that statistical mechanics is rather special in that it provides us with the first 
examples of fundamental laws of physics that can be derived as examples of 
inductive inference. Perhaps all laws of physics can be derived in this way. 

The level of these lectures is somewhat uneven. Some topics are fairly ad- 
vanced - the subject of recent research - while some other topics are very el- 
ementary. I can give two related reasons for including the latter. First, the 
standard education of physicists includes a very limited study of probability 
and even of entropy - maybe just a little about errors in a laboratory course, 
or maybe a couple of lectures as a brief mathematical prelude to statistical me- 
chanics. The result is a widespread misconception that these subjects are trivial 
and unproblematic - that the real problems of theoretical physics lie elsewhere, 
and that if your experimental data require analysis, then you have done the 
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wrong experiment. Which brings me to the second reason. It would be very 
surprising to find that the interpretations of probability and of entropy turned 
out to bear no relation to our understanding of statistical mechanics and quan- 
tum mechanics. Indeed, if the only notion of probability at your disposal is that 
of a frequency in a large number of trials you might be led to think that the en- 
sembles of statistical mechanics must be real, and to regard their absence as an 
urgent problem demanding an immediate solution - perhaps an ergodic solution. 
You might also be led to think that similar ensembles are needed in quantum 
theory and therefore that quantum theory requires the existence of an ensemble 
of parallel universes. Similarly, if the only notion of entropy available to you 
is derived from thermodynamics, you might end up thinking that entropy is a 
physical quantity related to heat and disorder, that it can be measured in the 
lab, and that therefore has little or no relevance beyond statistical mechanics. 

It is very worthwhile to revisit the elementary basics not because they are 
easy - they are not - but because they are fundamental. 

Many are the subjects that I have left out but wish I had included in these 
lectures. Some relate to inference proper - the assignment of priors, information 
geometry, model selection, and the theory of questions or inductive inquiry - 
while others deal with applications to the foundations of both classical and 
quantum physics. As a provisional remedy at the very end I provide a short 
and very biased list of suggestions for further reading. 

Acknowledgements: The points of view expressed here reflect much that I 
have learned from discussions with many colleagues and friends: C. Cafaro, N. 
Caticha, V. Dose, R. Fischer, A. Garrett, A. GifRn, M. Grendar, K. Knuth, R. 
Preuss, C. Rodriguez, J. Skilling, and C.-Y. Tseng. I hope they will not judge 
these lectures by those few instances where we have not yet managed to reach 
agreement. I would also like to express my special thanks to Julio Stern and 
to the organizers of MaxEnt 2008 for their encouragement to pull these notes 
together into some sort of printable form. 
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Chapter 1 

Inductive Inference 



The process of drawing conclusions from available information is called infer- 
ence. When the available information is sufficient to make unequivocal, unique 
assessments of truth we speak of making deductions: on the basis of a certain 
piece of information we deduce that a certain proposition is true. The method 
of reasoning leading to deductive inferences is called logic. Situations where the 
available information is insufficient to reach such certainty lie outside the realm 
of logic. In these cases we speak of making a probable inference, and the method 
of reasoning is probability theory. Alternative names are 'inductive inference' 
and 'inductive logic'. The word 'induction' refers to the process of using limited 
information about a few special cases to draw conclusions about more general 
situations. 

1.1 Probability 

The question of the meaning and interpretation of the concept of probability has 
long been controversial. Needless to say the interpretations offered by various 
schools are at least partially successful or else they would already have been 
discarded. But the different interpretations are not equivalent. They lead people 
to ask different questions and to pursue their research in different directions. 
Some questions may become essential and urgent under one interpretation while 
totally irrelevant under another. And perhaps even more important: under 
different interpretations equations can be used differently and this can lead to 
different predictions. 

Historically the frequentist interpretation has been the most popular: the 
probability of a random event is given by the relative number of occurrences of 
the event in a sufficiently large number of identical and independent trials. The 
appeal of this interpretation is that it seems to provide an empirical method 
to estimate probabilities by counting over the set of trials - an ensemble. The 
magnitude of a probability is obtained solely from the observation of many 
repeated trials and does not depend on any feature or characteristic of the 
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observers. Probabilities interpreted in this way have been called objective. This 
view dominated the fields of statistics and physics for most of the 19th and 20th 
centuries (see, e.g., [von Mises 57]). 

One disadvantage of the frequentist approach has to do with matters of rigor: 
what precisely does one mean by 'random'? If the trials are sufficiently identical, 
shouldn't one always obtain the same outcome? Also, if the interpretation is to 
be validated on the basis of its operational, empirical value, how large should 
the number of trials be? Unfortunately, the answers to these questions are 
neither easy nor free from controversy. By the time the tentative answers have 
reached a moderately acceptable level of sophistication the intuitive appeal of 
this interpretation has long been lost. In the end, it seems the frequentist 
interpretation is most useful when left a bit vague. 

A more serious objection is the following. In the frequentist approach the 
notion of an ensemble of trials is central. In cases where there is a natural 
ensemble (tossing a coin, or a die, spins in a lattice, etc.) the frequency inter- 
pretation seems natural enough. But for many other problems the constrTiction 
of an ensemble is at best highly artificial. For example, consider the probability 
of there being life in Mars. Are we to imagine an ensemble of Mars planets and 
solar systems? In these cases the ensemble would be purely hypothetical. It 
offers no possibility of an empirical determination of a relative frequency and 
this defeats the original goal of providing an objective operational interpretation 
of probabilities as frequencies. In yet other problems there is no ensemble at 
all: consider the probability that the nth digit of the number tt be 7. Are we to 
imagine alternative universes with different values for the number tt? It is clear 
that there a number of interesting problems where one suspects the notion of 
probability could be quite useful but which nevertheless lie outside the domain 
of the frequentist approach. 

According to the Bayesian interpretations, which can be traced back to 
Bernoulli and Laplace, but have only achieved popularity in the last few decades, 
a probability reflects the confidence, the degree of belief of an individual in the 
truth of a proposition. These probabilities are said to be Bayesian because of 
the central role played by Bayes' theorem a theorem which is actually due 
to Laplace. This approach enjoys several advantages. One is that the difficul- 
ties associated with attempting to pinpoint the precise meaning of the word 
'random' can be avoided. Bayesian probabilities are not restricted to repeat- 
able events; they allow us to reason in a consistent and rational manner about 
unique, singular events. Thus, in going from the frequentist to the Bayesian 
interpretations the domain of applicability and therefore the usefulness of the 
concept of probability is considerably enlarged. 

The crucial aspect of Bayesian probabilities is that different individuals may 
have different degrees of belief in the truth of the very same proposition, a 
fact that is described by referring to Bayesian probabilities as being subjective. 
This term is somewhat misleading because there are (at least) two views on 
this matter, one is the so-called subjective Bayesian or personalistic view (see, 
e.g., [Savage 72, Howson Urbach 93, Jeffrey 04]), and the other is the objective 
Bayesian view (see e.g. [Jeffreys 39, Jaynes 85, 03, Lucas 70]). For an excellent 
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introduction with a philosophical perspective see [Hacking 01]. According to 
the subjective view, two reasonable individuals faced with the same evidence, 
the same information, can legitimately differ in their confidence in the truth 
of a proposition and may therefore assign different probabilities. Subjective 
Bayesians accept that an individual can change his or her beliefs, merely on the 
basis of introspection, reasoning, or even revelation. 

At the other end of the Baycsian spectrum, the objective Bayesian view 
considers the theory of probabihty as an extension of logic. It is said then 
that a probability measures a degree of rational belief. It is assumed that the 
objective Baycsian has thought so long and hard about how probabilities arc 
assigned that no further reasoning will induce a revision of beliefs except when 
confronted with new information. In an ideal situation two different individuals 
will, on the basis of the same information, assign the same probabilities. 

Whether Bayesian probabilities are subjective or objective is still a matter 
of dispute. Our position is that they lie somewhere in between. Probabilities 
will always retain a "subjective" clement because translating information into 
probabilities involves judgments and different people will inevitably judge dif- 
ferently. On the other hand, not all probability assignments are equally useful 
and it is plausible that what makes some assignments better than others is that 
they represent or reflect some objective feature of the world. One might even 
say that what makes them better is that they provide a better guide to the 
"truth" . Thus, probabilities can be characterized by both subjective and ob- 
jective elements and, ultimately, it is their objectivity that makes probabilities 
useful. 

In fact we shall sec that while the subjective clement in probabilities can 
never be completely eliminated, the rules for processing information, that is, 
the rules for updating probabilities, are themselves quite objective. This means 
that the new information can be objectively processed and incorporated into 
our posterior probabilities. Thus, it is quite possible to continuously suppress 
the subjective elements while enhancing the objective elements as we process 
more and more information. 

1.2 Inductive reasoning 

We discussed how the study of macroscopic systems requires a general theory 
to allow us to carry out inferences on the basis of incomplete information and 
our first step should be to inquire what this theory or language for inference 
should be. The principle of reasoning that we will follow is simple, compelling, 
and quite common in science [Skilling 89] : 

If a general theory exists, it must apply to special cases. 

If a certain special case happens to be known then this knowledge can be 
used to constrain the general theory: all candidate theories that fail to 
reproduce the known example are discarded. 
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If a sufficient number of special cases is known then the general theory 
might be completely determined. 

The method allows us to extrapolate from a few special cases where we know 
what to expect, to more general cases where we did not. This is a method for 
induction, for generalization. Of course, it may happen that there are too many 
constraints, in which case there is no general theory that reproduces them all. 

Philosophers have a name for such a method: they call it eliminative in- 
duction [Earman 92]. On the negative side, the Principle of Eliminative 
Induction (PEI), like any other form of induction, is not guaranteed to work. 
On the positive side, the PEI adds an interesting twist to Popper's scientific 
methodology. According to Popper scientific theories can never be proved right, 
they can only be proved false; a theory is corroborated only to the extent that 
all attempts at falsifying it have failed. Eliminative induction is fully compati- 
ble with Popper's notions but the point of view is just the opposite. Instead of 
focusing on failure to falsify one focuses on success: it is the successful falsifica- 
tion of all rival theories that corroborates the surviving one. The advantage is 
that one acquires a more explicit understanding of why competing theories are 
eliminated. 

This inductive method will be used several times. First in chapter 2 to show 
that if a general theory of inference exists, then it must coincide with the usual 
theory of probability. In other words, we will show that degrees of belief, those 
measures of plausibility that we require to do inference, should be manipulated 
and calculated using the ordinary rules of the calculus of probabihties and there- 
fore that probabilities can be interpreted as degrees of belief [Cox 46, Jaynes 
57a, 03]. 

But with this achievement, enormous as it is, we do not yet reach our fi- 
nal goal. The problem is that what the rules of probability theory will allow 
us to do is to assign probabilities to some "complex" propositions on the ba- 
sis of the probabihties that have been previously assigned to other, perhaps 
more "elementary" propositions. The issue of how to assign probabilities to the 
elementary propositions is not addressed. 

Historically the first partial solution to this problem was suggested by James 
Bernoulli (1713). The idea is simple: in those situations where there are several 
alternatives that can be enumerated and counted, and where one has no reason 
to favor one over another, the alternatives should be deemed equally probable. 
The equality of the degrees of belief reflects the symmetry of one's state of 
knowledge or, rather, of ignorance. This mode of reasoning has been called the 
'Principle of Insufficient Reason^ and is usually associated with the name of 
Laplace (1812). 

The principle has been particularly successful in dealing with situations 
where there is some positive, sufficient reason to suspect that the various al- 
ternatives should be considered equally likely. For example, in certain games 
of chance the symmetry among possible outcomes is attained on purpose, by 
construction. These games are special because they are deliberately designed 
so that information about previous outcomes is irrelevant to the prediction of 
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future outcomes and the symmetry of our state of ignorance about the future 
is very robust. 

The range of applications of Laplace's principle is, however, limited. There 

are situations where it is not clear what 'equally likely' means. For example, it 
might not be possible to count the alternatives or maybe the possible outcomes 
are distributed over continuous ranges. Also, there are situations where there is 
information leading one to prefer some alternatives over others; how can such 
information be incorporated in a systematic way? One needs a method that 
generalizes Laplace's principle. 

Progress toward this goal came from an unexpected direction. While investi- 
gating the capacity of communication channels to transmit information Shannon 
came to appreciate the need for a quantitative measure of the notion of "amount 
of missing information" or the "amount of uncertainty" in a probability distri- 
bution. In 1948 he succeeded in finding such a measure and thereby initiated 
the field of information theory [Shannon 48]. 

As we will see in chapter 4 Shannon's argument is a second application of 
the induction principle above: A general theory, if it exists at all, must apply to 
special cases. He argued that in order to qualify as a measure of ignorance or of 
missing information a quantity S would have to satisfy some reasonable condi- 
tions - the Shannon axioms - and these conditions were sufficiently constraining 
to determine the quantity S uniquely: There is only one way to measure the 
amount of uncertainty in a probability distribution. It was rather surprising that 
the expression that Shannon obtained for S in communication theory coincided 
with expressions that had previously been used by Boltzmann and by Gibbs 
to represent entropy in the very different context of statistical mechanics and 
thermodynamics. This coincidence led Shannon to choose the name 'entropy' 
for his quantity S. Somewhat later, however, Brillouin and Jaynes realized that 
the similarity of Shannon's entropy with Gibbs' entropy could not be a mere co- 
incidence and thus began a process that would radically alter our understanding 
of the thermodynamical entropy of Glausius. [Brillouin 52, Jaynes 57b] 

The crucial contribution of Jaynes was the insight that the Shannon deriva- 
tion was not limited to information in communication channels, but that the 
same mathematics can be applied to information in general. It establishes a basis 
for a general method of inference that includes Laplace's principle of insufficient 
reason as a special case. In fact, it became clear that on a purely intuitive basis 
Boltzmann and Gibbs had already found and had made extensive use of this 
method in statistical mechanics. 

With the Boltzmann-Gibbs-Jaynes method we can revisit the question of 
how to assign those probabilities that will be used as the starting point for the 
calculation of all others. The answer is simple: among all possible probability 
distributions that satisfy the constraints implied by the limited available infor- 
mation we select that particular distribution that reflects maximum ignorance 
about those aspects of the problem about which nothing is known. What else 
could we do? It seems this is the only intellectually honest way to proceed. And 
the procedure is mathematically clear: since ignorance is measured by entropy 
the desired probability distribution is obtained by maximizing the entropy sub- 
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ject to whatever conditions are known to constrain the system. This is called 
the Method of Maximum Entropy and it is usually abbreviated as MaxEnt. 

But the procedure is not without its problems. These may, to some, seem 
relatively minor, but one may reasonably argue that any problem of principle 
is necessarily a major problem. For example, the Shannon axioms refer to 
discrete probability distributions rather than continuous ones, and generalizing 
his measure of uncertainty is not altogether straightforward. Another, perhaps 
more serious problem, is that the axioms themselves may be self-evident to 
some but not to others: do the Shannon axioms really codify what we mean 
by uncertainty? Are there other measures of uncertainty? Indeed, others have 
been proposed. Thus, despite its obvious success, in the eyes of many, the 
MaxEnt method remains controversial and several variations on its justification 
have been proposed. 

In chapter 6 we present an extension of the method of maximum entropy 
(which we will abbreviate ME to distinguish it from the older MaxEnt) which 
derives from the work of Shore and Johnson. They point out what is perhaps 
the main drawback of the Shannon-Jaynes approach; it is indirect. First one 
finds how to measure amount of uncertainty and then one argues that the only 
unbiased way to incorporate information into a probability distribution is to 
maximize this measure subject to constraints. The procedure can be challenged 
by arguing that, even granted that entropy measures something, how sure can 
we be this something is uncertainty, ignorance? Shore and Johnson argue that 
what one really wants is a consistent method to process information directly, 
without detours that invoke questionable measures of uncertainty. 

A third application of the general inductive method - a general theory, if it 
exists at all, must apply to special cases [Skilling 88] - yields the desired proce- 
dure: There is a unique method to update from an old set of beliefs codified in a 
prior probability distribution into a new set of beliefs described by a new, poste- 
rior distribution when the information available is in the form of a constraint on 
the family of acceptable posteriors. The updated posterior distribution is that 
of maximum "relative" entropy. The axioms of the ME method arc, hopefully, 
more self-evident: They reflect the conviction that what was learned in the past 
is important and should not be frivolously ignored. The chosen posterior dis- 
tribution should coincide with the prior as closely as possible and one should 
only update those aspects of one's beliefs for which corrective new evidence has 
been supplied. Furthermore, since the new axioms do not tell us what and how 
to update, they merely tell us what not to update, they have the added bonus 
of maximizing objectivity - there are many ways to change something but only 
one way to keep it the same. [Caticha 03,Caticha Giffin 06, Caticha 07] 

This alternative justification for the method of maximum entropy turns out 
to be directly applicable to continuous distributions, and it establishes the value 
of the concept of entropy irrespective of its interpretation in terms of heat, or 
disorder, or uncertainty. In this approach entropy is purely a tool for consistent 
reasoning; strictly, it needs no interpretation. Perhaps this is the reason why 
the meaning of entropy has turned out to be such an elusive concept. 



Chapter 2 

Probability 



Our goal is to establish the theory of probability as the general theory for 
reasoning on the basis of incomplete information. This requires us to tackle 
two different problems. The first problem is to figure out how to achieve a 
quantitative description of a state of knowledge. Once this is settled we address 
the second problem of how to update from one state of knowledge to another 
when new information becomes available. 

Throughout we will assume that the subject matter - the set of statements 
the truth of which we want to assess ~ has been clearly specified. This question 
of what it that we are actually talking about is much less trivial than it might 
appear at first sight|^ Nevertheless, it will not be discussed further. 

The first problem, that of describing or characterizing a state of knowledge, 
requires that we quantify the degree to which we believe each proposition in 
the set is true. The most basic feature of these beliefs is that they form an 
interconnected web that must be internally consistent. The idea is that in 
general the strengths of one's beliefs in some propositions are constrained by 
one's beliefs in other propositions; beliefs are not independent of each other. For 
example, the belief in the truth of a certain statement a is strongly constrained 
by the belief in the truth of its negation, not-a: the more I believe in one, 
the less I believe in the other. As we will see below, the basic desiderata for 
such a scheme, which are expressed in the Cox axioms, [Cox 46] lead to a 
unique formalism in which degrees of belief are related to each other using the 
standard rules of probability theory. Then we explore some of the consequences. 
For experiments that can be repeated indefinitely one recovers standard results, 
such as the law of large numbers, and the connection between probability and 
frequency. 

The second problem, that of updating from one consistent web of beliefs 
to another when new information becomes available, will be addressed for the 
special case that the information is in the form of data. The basic updating 

^Consider the example of quantum mechanics: Are we talking about particles, or about 
experimental setups, or both? Are we talking about position variables, or about momenta, or 
both? Or neither? Is it the position of the particles or the position of the detectors? 
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strategy reflects the conviction that what we learned in the past is vahiable, 
that the web of beUefs should only be revised to the extent required by the 
data. We will see that this principle of minimal updating leads to the uniquely 
natural rule that is widely known as Bayes' theorem. (More general kinds of 
information can also be processed using the minimal updating principle but they 
require a more sophisticated tool, namely relative entropy. This topic will be 
extensively explored later.) As an illustration of the enormous power of Bayes' 
rule we will briefly explore its application to data analysis. 

2.1 Consistent reasoning: degrees of belief 

We discussed how the study of physical systems in general requires a theory 
of inference on the basis of incomplete information. Here we will show that a 
general theory of inference, if it exists at all, coincides with the 'usual theory 
of probability. We will show that the quantitative measures of plausibility or 
degrees of belief that we introduce as tools for reasoning should be manipulated 
and calculated using the ordinary rules of the calculus of probabilities. Therefore 
probabilities can be interpreted as degrees of belief. 

The procedure we follow differs in one remarkable way from the traditional 
way of setting up physical theories. Normally one starts with the mathematical 
formalism, and then one proceeds to try to figure out what the formalism could 
possibly mean, one tries to append an interpretation to it. This is a very difficult 
problem; historically it has affected not only statistical physic;H what is the 
meaning of probabilities and of entropy - but also quantum theory - what 
is the meaning of wave functions and amplitudes. Here we proceed in the 
opposite order, we first decide what we are talking about, degrees of belief 
or plausibility (we use the two expressions interchangeably) and then we design 
rules to manipulate them; we design the formalism, we construct it to suit 
our purposes. The advantage of this approach is that the issue of meaning, of 
interpretation, is settled from the start. 

Before we proceed further it may be important to emphasize that the degrees 
of belief discussed here are those held by an idealized rational agent that would 
not be subject to the practical limitations under which we humans operate. 
We discuss degrees of rational belief and not the irrational and inconsistent 
beliefs that real humans seem to hold. We are concerned with the ideal optimal 
standard of rationality that we humans ought to attain at least when discussing 
scientific matters. 

Any suitable measure of belief must allow us to represent the fact that given 
any two statements a and b one must be able to describe the fact that either a 
is more plausible than b, or a is less plausible than b, or else a and b are equally 
plausible. That this is possible is implicit in what we mean by 'plausibility'. 
Thus we can order assertions according to increasing plausibility: if a statement 
a is more plausible than 6, and b is itself more plausible than another statement 
c, then a is more plausible than c. Since any transitive ordering, such as the one 
just described, can be represented with real numbers, we are led to the following 
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requirement: 

Degrees of rational belief (or, as we shall later call them, probabilities) 
are represented by real numbers. 

The next and most crucial requirement is that whenever a degree of beHef can 
be computed in two different ways the two results must agree. 

The assignment of degrees of rational belief must be consistent. 

Otherwise we could get entangled in confusing paradoxes: by following one 
computational path we could decide that a statement a is more plausible than 
a statement b, but if we were to follow a different path wc could conclude the 
opposite. Consistency is the crucial requirement that eliminates vagueness and 
transforms our general qualitative statements into precise quantitative ones. 

Our general theory of inference is constructed using the inductive method 
described in the previous chapter: If a general theory exists, then it must re- 
produce the right answers in those special cases where the answers happen to 
be known; these special cases constrain the general theory; given enough such 
constraints, the general theory is fully determined. 

Before we write down the special cases that will play the role of the axioms 
of probability theory wc should introduce a convenient notation. A degree 
of plausibility is a real number that we will assign to a statement a on the 
basis of some information that we have and will obviously depend on what 
that information actually is. A common kind of information takes the form 
of another statement b which is asserted to be true. Therefore, a degree of 
plausibility is a real number assigned to two statements a and b, rather than 
just one. Our notation should reflect this. Let P{a\b) denote the plausibility that 
statement a is true provided we know b to be true. P{a\b) is read 'the degree of 
plausibility (or, later, the probability) of a given b\ P{a\b) is commonly called 
a conditional probability (the probability of a given that condition b holds). 
When b turns out to be false, we shall regard P{a\b) as undefined. Although 
the notation P{a\b) is quite convenient we will not always use it; we will often 
just write P (a) omitting the statement b, or wc might even just write P. It is, 
however, important to realize that degrees of belief and probabilities are always 
conditional on something even if that something is not explicitly stated. 

More notation: For every statement a there exists its negation not-a, which 
will be denoted with a prime, a'. If a is true, then a' is false and vice versa. 
Given two statements ai and 02 we can form their conjunction 'ai and 02' which 
we will denote it as 0102. The conjunction is true if and only if both ai and 02 
are true. Given ai and 02, we can also form their disjunction 'ai or 02'- The 
disjunction will be denoted by Oi + 02 and it is true when either ai or 02 or 
both are true; it is false when both ai and 02 are false. 

Now we proceed to state the axioms [Cox 46, Jaynes 03]. 
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2.2 The Cox Axioms 

The degrees of belief or plausibility we assign to a statement a and to its negation 
a' are not independent of each other. The more plausible one is, the less plausible 
the other becomes; if one increases we expect the other to decrease and vice- 
versa. This is expressed by our first axiom. 

Axiom 1. The plausibility of not-a is a monotonic function of the plausibility 
of a, 

P{a'\b)^f{P{a\b)). (2.1) 

At this point we do not know the precise relation between P(a\b) and P{a'\b), 
we only know that some such function / must exist. 

The second axiom expresses the fact that a measure of plausibility for a 
complex statement such as the conjunction "ai and 02" , must somehow depend 
on the separate plausibilities of ai and of 02. We consider it "self-evident" that 
the plausibility that both ai and a2 are simultaneously true, P(aia2\b), can be 
analyzed in stages: In order for 0102 to be true it must first be the case that 
ai is itself true. Thus, P{aia2\b) must depend on P(ai\b). Furthermore, once 
we have established that oi is in fact true, in order for 0102 to be true, it must 
be the case that 02 is also true. Thus, P{aia2\b) must depend on P{a2\aib) as 
well. This argument is carried out in more detail in [Tribus 69]. Therefore, our 
second axiom is 

Axiom 2. The plausibility P(aia2\b) of a conjunction aia2, is determined once 
we specify the plausibility P(ai\b) of ai and the plausibility P{a2\aib) of 
02 given fli. 

What this means is that P{aia2\b) must be calculable in terms of P{ai\b) and 
P(a2\aib): the second axiom asserts that there exists a function g such that 

P(aia2|fe) = g {P{a,\b), P{a2\aib)) . (2.2) 

Remarkably this is all we need! Note the qualitative nature of these axioms: 
what is being asserted is the existence of some unspecified functions / and g 
and not their specific quantitative mathematical forms. Furthermore, note that 
the same / and g apply to any and all propositions. This reflects our desire 
to construct a single theory of universal applicability. It also means that the 
axioms represent a huge number of known special cases. 

At this point the functions / and g are unknown, but they are not arbitrary. 
In fact, as we shall see below, the requirement of consistency is very constraining. 
For example, notice that since 0102 = 0,20,1, in |2.2| the roles of oi and 02 may 
be interchanged, 

P{aia2\b)^g{P{a2\b),P{ai\o2b)). (2.3) 
Consistency requires that 



g (P(ai|6), F(a2|ai6)) - g iPia2\b), P{ai\a2b)) . (2.4) 
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We will have to check that this is indeed the case. As a second example, since 
a" = a, it must be the case that 

P{a\b) = P{a"\b) = / {P{a'\b)) = f [f {P{a\b))] . (2.5) 

The plausibility P(a\b) is just a number, call it u, this can be written as 

f{f{u))^u. (2.6) 

These two constraints are not at this point helpful in fixing the functions / and 
g. But the following one is. 



2.3 Regraduation: the Product Rule 
2.3.1 Cox's first theorem 

A consistency constraint that follows from the associativity property of the 
conjunction goes a long way toward fixing the acceptable forms of the function 
g. The constraint is obtained by noting that since (ab) c = a (be), we have two 
ways to compute P {abc\d). Starting from 

P [{ab) c\d] = P[a (be) \d] , (2.7) 

we get 

g [P {ab\d) , P {c\abd)] = g [P {a\d) , P {bc\ad)] (2.8) 

and 

g [g (P {a\d) , P {b\ad)) , P {c\abd)] =g[P {a\d) , g {P {b\ad) , P {c\bad))] . (2.9) 



Writing P {a\d) — u, P {b\ad) — v, and P {c\abd) = w, the "associativity" 
constraint is 

g{g{u,v),w) = g{u,g{v,w)) . (2.10) 



It is quite obvious that the functional equation eq.(2.10l has an infinity of 
solutions. Indeed, by direct substitution one can easily check that functions of 
the form 

g{u,v)) = G-'[G{u)G{v)] (2.11) 

are solutions for any invertible (and therefore monotonic) function G{u). What 
is not so easy to prove is that this is the general solution. 

Associativity Theorem: Given any function g{u, v) that satisfies the associa- 
tivity constraint, eq.( 2.10[ ), one can construct another monotonic function G{u) 
such that 

G{g{u,v))^G{u)G{v). (2.12) 

Cox's proof of this theorem is somewhat lengthy and is relegated to the next 
subsection. 
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The significance of this result becomes apparent when one rewrites it as 

G [P{ab\c)] = G [P{a\c)] G [P{b\ac)] (2.13) 

and realizes that there was nothing particularly special about the original as- 
signment of real numbers P{a\c), P{b\ac), and so on. Their only purpose was to 
provide us with a ranking, an ordering of propositions according to how plau- 
sible they are. Since the function G{u) is monotonic, the same ordering can be 
achieved using a new set positive numbers 

p{a\c)''=' G[P{a\c)], p{b\ac) ""^^ G [P{b\ac)] , ... (2.14) 

instead of the old. The advantage of using these 'regraduated' plausibilities is 
that the plausibility of ab can be calculated in terms of the plausibilities of a 
and of b given a in a particularly simple way: it is just their product. Thus, 
while the new numbers are neither more nor less correct than the old, they are 
just considerably more convenient. The theorem can be rephrased as follows. 
Cox's First Regraduation Theorem: Once a consistent representation of 
the ordering of propositions according to their degree of plausibility has been 
set up by assigning a real number P(a\b) to each pair of propositions a and 
b one can always find another equivalent representation by assigning positive 
numbers p{a\c) that satisfy the product rule 

p{ab\c) = p{a\c)p{b\ac). (2.15) 

Perhaps one can make the logic behind this regraduation a little bit clearer 
by considering the somewhat analogous situation of introducing the quantity 
temperature as a measure of degree of "hotness" . Clearly any acceptable mea- 
sure of "hotness" must reflect its transitivity - if a is hotter than b and b is 
hotter than c then a is hotter than c; thus, temperatures are represented by 
real numbers. But the temperature scales are so far arbitrary. While many 
temperature scales may serve equally well the purpose of ordering systems ac- 
cording to their hotness, there is one choice - the absolute or Kelvin scale - that 
turns out to be considerably more convenient because it simplifies the mathe- 
matical formalism. Switching from an arbitrary temperature scale to the Kelvin 
scale is one instance of a convenient regraduation. (The details of temperature 
regraduation are given in chapter 3.) 

On the basis of plain common sense one would have expected g{u, v) to 
be monotonic in both its arguments. Consider a change in the first argument 
P(ai|6) while holding the second P{a2\aib) fixed. Since a strengthening the 
belief in ai can only strengthen the belief in aia2 we require that a change in 
P{ai\b) should yield a change in P{aia2\b) of the same sign. It is therefore a 



reassuring check that the product rule eq.(2.15) behaves as expected 



2.3.2 Proof of the Associativity Theorem 

Understanding the proof that eq.( |2.12| is the general solution of the associativity 



constraint, eq.(2.10l, is not necessary for understanding other topics in this 
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book. This section may be skipped on a first reading. The proof given below, 
due to Cox, takes advantage of the fact that our interest is not just to find the 
most general solution but rather that we want the most general solution under 
the restricted circumstance that the function g is to be used for the purpose of 
inference. This allows us to impose additional constraints on g. 

We will assume that the functions g are continuous and twice differentiable. 
Indeed inference is quantified common sense and if the function g had turned out 
to be non-differentiable serious doubts would be cast on the legitimacy of the 
whole scheme. Furthermore, common sense also requires that g(u,v) be mono- 
tonic increasing in both its arguments. Consider a change in the first argument 
P{ai\b) while holding the second P{a2\aib) fixed. Since a strengthening of one's 
belief in oi must be reflected in a corresponding strengthening in ones's belief in 
aia2 we require that a change in P{ai\b) should yield a change in P{aia2\b) of 
the same sign. An analogous line of reasoning leads one to impose that g{u, v) 
must be monotonic increasing in the second argument as well, 

^^("'")>0 and ^l^>0. (2.16) 



Let 



du dv 



r'^g{u,v) and s =^g(w,w), (2-17) 



and 



, dcf dg{u,v) dcf dg{u,v) 

gi[u,v) = — — — >0 and g2[u,v) = — — — > 0. (2.18) 



Then eq.(2.10l and its derivatives with respect to u and v are 

g{r,w)^giu,s), (2.19) 

9i{r,w)gi{u,v) = gi{u,s), (2.20) 

and 

9i{r,w)g2{u,v) = g2{u, s)gi{v,w). (2.21) 

Eliminating gi{r, w) from these last two equations we get 

K{u,v)^ K{u,.s)gi{v,w). (2.22) 



where 

i^(.,.) = ^. (2.23) 



Multiplying eq.(2.22) by K{v,w) we get 



K{u, v)K{v, w) ~ K{u, s)g2{v, w) 



(2.24) 
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Differentiating the right hand side of eq.(2.24l with respect to v and comparing 



with the derivative of eq.(2.22l with respect to w, we have 

o o o 

— {K {u, s) 02 {v, w)) ^T^{K {u, s) gi {v, w)) ^ ^ {K {u, v)) = 0. (2.25) 
ov aw aw 



Therefore 



or, 



dv 



{K {u,v)K{v,w)) =0, 



1 dK{u,v) 



1 dK {v, w) def 



h{v) 



(2.26) 
(2.27) 



K(u,v) dv K(v,w) dv 

Integrate using the fact that if > because both gi and g2 are positive, we get 

(2.28) 



and also 



so that 



K{u, v) = K{u, 0) exp / h{v')dv' , 
Jo 



K{v,w) = K{0,w) exp- / h{v')dv', 

Jo 



K (u, v) = a 



H{u) 



H{vy 

where a — if(0, 0) is a constant and H{u) is the positive function 



H{u) =^ exp 



h{u')du' 



> 0. 



On substituting back into eqs.(2.22l and (2.241 we get 



H{v) 



and g2{v, w) = a 



H{s) 
H{w)' 



(2.29) 



(2.30) 



(2.31) 



(2.32) 



Next, use s — g{v,w), so that 

ds — gi{v, w)dv + 52 (w, w)dw. 



Substituting (2.32) we get 



ds dv dw 
a- 



H{s) H{v) H{w)' 
This is easily integrated. Let 

du' 



G{u) = G (0) exp 



Hi^')J' 



(2.33) 



(2.34) 



(2.35) 
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so that du/H{u) = dG{u)/G{u). Then 



Gig{v,w))^G{v) (w) 



(2.36) 



where a multiphcative constant of integration has been absorbed into the con- 
stant G (0). Applying this function G twice in eq.( |2.10[ ) we obtain 



so that a = 1, 



G(u)G"(w)G"(w;) = G(w)G"(v)G" (w), 



G{g{v,w))^G{v)G{w), 



(2.37) 



(2.38) 



(The second possibility a = is discarded because it leads to g{u, v) = u which 



is not useful for inference.) This completes our proof eq.(2.12l is the general 
solution of eq. (2.101: Given any g{u,v) that satisfies eq. (2.101 one can construct 
the corresponding G(u) using eqs. (2. 23 1, (2.271, (2.311, and(2.35l. Furthermore, 
since G(u) is an exponential its sign is dictated by the constant G (0) which 
is positive because the right hand side of eq.(2.38) is positive. Finally, since 
H{u) > 0, eq. (2.31), the regraduating function G(u) is a monotonic function 
of its variable u. 



2.3.3 Setting the range of degrees of belief 

Degrees of belief range from the extreme of total certainty that an assertion is 
true to the opposite extreme of total certainty that it is false. What numerical 
values should we assign to these extremes? 

Let pt and pp he the numerical values assigned to the (regraduated) plausi- 
bilities of propositions which are known to be true and false respectively. Notice 
that the extremes should be unique. There is a single px and a single pp- The 
possibility of assigning two different numerical values, for example pxi and pt2, 
to propositions known to be true is ruled out by our desire that degrees of 
plausibility be ordered. 

The philosophy behind regraduation is to seek the most convenient repre- 
sentation of degrees of belief in terms of real numbers. In particular, we would 
like our regraduated plausibilities to reflect the fact that if b is known to be true 
then we believe in ab to precisely the same extent as we believe in a, no more 
and no less. This is expressed by 



piab\b) = pia\b) . 



On the other hand, using the product rule eq.(2.15l we get 



p{ab\b) = p{b\b)p{a\bb) — pTp{a\b) . 



Comparing eqs.(2.39) and (2.401 we get 

Pt = 1 



(2.39) 
(2.40) 

(2.41) 



Thus, the value of px is assigned so that eq.(2.39l holds: 
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Belief that a is true is represented by p{a) ~ 1. 

For the other extreme value, pp, which represents impossibihty, consider the 
plausibility of ab' given b. Using the product rule we have 

p{ab'\b) = p{a\b)p{b'\ab) . (2.42) 

But p{ab'\b) — pf and p(b'\ab) — pp. Therefore 

PF^p{a\b)pF. (2.43) 

Again, this should hold for arbitrary a. Therefore either p^? or cxd, either 
value is fine. (The value — oo is not allowed; negative values of p{a\b) would 
lead to an inconsistency.) We can either choose plausibilities in the range [0, 1] 
so that a higher p reflects a higher degree of belief or, alternatively, we can 
choose 'implausibilities' in the range [1, oo) so that a higher p reflects a lower 
degree of belief. Both alternatives are equally consistent and correct. The usual 
convention is to choose the former. 

Belief that a is false is represented by p(a) — 0. 

The numerical values assigned to pt and pp follow from a particularly con- 
venient regraduation that led to the product rule. Other possibilities are, of 



course, legitimate. Instead of eq.(2.14l we could for example have regraduated 



plausibilities according to p{a\c) '= CG [P{a\c)] where C is some constant. Then 
the product rule would read Cp{ab\c) — p{a\c)p{b\ac) and the analysis of the 
previous paragraphs would have led us to pr = C and pp — or oo. The choice 
C — 100 is quite common; it is implicit in many colloquial uses of the notion of 
probability, as for example, when one says 'I am 100% sure that...'. Notice, in- 
cidentally, that within a frequentist interpretation most such statements would 
be meaningless. 



2.4 Further regraduation: the Sum Rule 
2.4.1 Cox's second theorem 

Having restricted the form of g considerably we next study the function / by 
requiring its compatibility with g. It is here that we make use of the constraints 
(2.4 1 and (2.6 1 that we had found earlier. 

Consider plausibilities P that have gone through a first process of regradu- 
ation so that the product rule holds, 

P{ab\c) = P{a\c)P{b\ac) = P{a\c)f {P{b'\ac)) (2.44) 

but P{ab'\c) = P{a\c)P{b'\ac), then 

P{ab\c) = P{a\c)f (^^^^) ■ (2.45) 
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But P(ab\c) is symmetric in ah = ha. Therefore 



P{a\c)f 



P{ab'\c) 
P{ci\c) 



Pib\c)f 



P{a'b\c) 



(2.46) 



This must hold irrespective of the choice of a, b, and c. In particular suppose 
that b' — ad. On the left hand side P{ab'\c) = P{b'\c) because aa = a. On the 
right hand side, to simplify P{a'b\c) we note that a'b' — a' ad is false and that 
a'b' — [a + b)' . (In order for a + 6 to be false it must be the case that both a is 
false and b is false.) Therefore a + & is true: either a is true or b is true. If b is 
true then a'b — a' . If a is true both a' and a'b are false which means that we 
also get a'b = a'. Therefore on the right hand side P{a'b\c) = P{a'\c) and we 
get 



P{a\c)f 



P{a\c) 



= P(6|c)/ 



P{b\c) 



(2.47) 



Writing P {a\c) — u, and P {b\c) — v, and P {c\abd) = w, the "compatibility" 
constraint is 



f{v) 



= vf 



/(") 



(2.48) 



We had earlier seen that certainty is represented by 1 and impossibility by 0. 
Note that when u — I, using /(I) — and /(O) — 1, we obtain f[f{v)] = v. 
Thus, eq.(2.6) is a special case of (2.481. 

Compatibility Theorem: The function f{u) that satisfies the compatibility 
constraint eq.( 2.48[ ) is 



/(u) = (l-u")^/" or w" + /"(u) = l. 



(2.49) 



where a is a constant. 

It is easy to show that eq.(2.49) is a solution - just substitute. What is con- 
siderably more difficult is to show that it is the general solution. The proof is 
given in the next subsection. 

As a result of the first theorem we can consider both u and f{u) positive. 
Therefore, for a > impossibility must be represented by 0, while for a < 
impossibility should be represented by oo. 

The significance of the solution for / becomes clear when eq.(2.49l is rewrit- 
ten as 

[Pia\br + [Pia'\br^l, (2.50) 



and the product rule eq.(2.44) is raised to the same power a, 

[P{ab\cr = [P{a\c)r[P{b\acr. (2.51) 

This shows that, having regraduated plausibilities once, we can simplify the 
solution (2.50 1 considerably by regraduating a second time, while still preserving 

(2.52) 



the product rule. This second regraduation is 



p{a\b) [Pia\. 
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Cox's Second Regraduation Theorem: Once a consistent representation of 
the ordering of propositions according to their degree of plausibility has been 
set up in such a way that the product rule holds, one can regraduate further and 
find an equivalent and more convenient representation that assigns plausibilities 
p{a\b) satisfying both the sum rule, 

p{a\b)+p{a'\b) = l, (2.53) 

and the product rule, 

p{ab\c) = p{a\c)p{b\ac). (2.54) 

These new, conveniently regraduated degrees of plausibility will be called 
probabilities, positive numbers in the interval [0, 1] with certainty represented by 
1 and impossibility by 0. From now on there is no need to refer to plausibilities 
again; both notations, lower case p as well as upper case P will be used to refer 
to the regraduated probabilities. 



2.4.2 Proof of the Compatibility Theorem 



The contents of this section is not essential to understanding other topics in this 
book. It may be skipped on a first reading. 

Just as in our previous consideration of the constraint imposed by associa- 
tivity on the function g, since the function / is to be used for the purpose of 
inference we can assume that it is continuous and twice differentiable. Further- 
more, once we have gone through the first stage of regraduation, and plausi- 



bilities satisfy the product rule eq.(2.15), common sense also requires that the 



function f{u) be monotonic decreasing. 



dfju) 
du 



< for < u < 1 , 



with extreme values such that /(O) = 1 and /(I) — 0. 

The first step is to transform the functional equation (2.48) into an ordinary 
diS^erential equation. Let 



dof / (v) dcf / (u) 

r — and s = . 



(2.55) 



and substitute into eq.(2.48l, 



ufir) = vf{s) 



(2.45) 



Next differentiate eq.(2.48l with respect to u, to v, and to u and v, to get (here 
primes denote derivatives) 



(2.56) 



m - sf'is) ^ nr)f'{v), 



(2.57) 
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and 



-r{s)f{u) = -/"(r)/'H. 

V u 



Multiply eq.(2.48l by eq.(2.58), 



s/"(s)/'H/(s) = r/"(r)/'(z;)/(r), 



(2.58) 



(2.59) 



and use eqs.(2.56l and (2.571 to eliminate f'{u) and f'{v). After rearranging 
one gets, 

sns)f{.s) ^ rf"{r)f{r) 
f'{s) im - sfis)] nr) [/(r) - r/'(r)] ' ^ ' ^ 

Since the left side does not depend on r, neither must the right side; both sides 
must actually be constant. Call this constant k. Thus, the problem is reduced 
to a differential equation. 



rf"{r)f{r) 
Multiplying by dr/rff gives 



fc/'W [f{r) - rf{r)] 



f 



= k 



dr 
r 



f 



Integrating twice gives 



fir) 



(2.61) 

(2.62) 
(2.63) 



(Ar"+B)^/" 

where A and B are integration constants and a — 1 + fc. Substituting back into 

on 
-1. 



eq. (2.48 ) allows us, after some simple algebra to determine one of the integration 
constants, B = A^, while substituting into eq.(2.6) yields the other, A 



This concludes the proof. 



2.5 Some remarks on the sum and product rules 

2.5.1 On meaning, ignorance and randomness 

The product and sum rules can be used as the starting point for a theory of 
probability: Quite independently of what probabilities could possibly mean, 
we can develop a formalism of real numbers (measures) that are manipulated 



according to eqs.(2.53l and (2.54). This is the approach taken by Kolmogorov. 
The advantage is mathematical clarity and rigor. The disadvantage, of course, 
is that in actual applications the issue of meaning, of interpretation, turns out 
to be important because it affects how and why probabilities are used. 

The advantage of the approach due to Cox is that the issue of meaning is 
clarified from the start: the theory was designed to apply to degrees of belief. 
Consistency requires that these numbers be manipulated according to the rules 
of probability theory. This is all we need. There is no reference to measures of 
sets or large ensembles of trials or even to random variables. This is remark- 
able: it means that we can apply the powerful methods of probability theory 
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to thinking and reasoning about problems where nothing random is going on, 
and to single events for which the notion of an ensemble is either absurd or at 
best highly contrived and artificial. Thus, probability theory is the method for 
consistent reasoning in situations where the information available might be in- 
sufficient to reach certainty: probability is the tool for dealing with uncertainty 
and ignorance. 

This interpretation is not in conflict with the common view that probabil- 
ities are associated with randomness. It may, of course, happen that there is 
an unknown influence that affects the system in unpredictable ways and that 
there is a good reason why this influence remains unknown, namely, it is so com- 
plicated that the information necessary to characterize it cannot be supplied. 
Such an influence we call 'random'. Thus, being random is just one among 
many possible reasons why a quantity might be uncertain or unknown. 

2.5.2 The general sum rule 



From the sum and product rules, eqs.(2.53l and (2.541 we can easily deduce a 
third one: 

Theorem: The probability of a disjunction (or) is given by the sum rule 

p{a + b\c) ^p{a\c) + p{b\c) - p{ab\c) . (2.64) 

The proof is straightforward. Use (a + b)' = a'b', (for a + 6 to be false both a 
and b must be false) then 

p{a + b\c) = l- p (a'6'|c) = l-p{a'\c)p (6'|a'c) = 

l-p{a'\c){l~p{b\a'c))^p{a\c)+p{a'b\c)^p{a\c)+p{b\c)p{a'\bc) = 

p (a|c) + p{b\c) (1 — _p ia\bc)) = p{a\c) + p{b\c) — p{ab\c). 

These theorems are rather obvious on the basis of the interpretation of a 
probability as a frequency or as the measure of a set. This is conveyed graphi- 
cally in a very clear way by Venn diagrams (see fig.2.1). 

2.5.3 Independent and mutually exclusive events 

In special cases the sum and product rules can be rewritten in various useful 
ways. Two statements or events a and b are said to be independent if the 
probability of one is not altered by information about the truth of the other. 
More specifically, event a is independent of b (given c) if 

p{a\bc) =p{a\c) . (2.65) 

For independent events the product rule simplifies to 

p{ab\c) — p{a\c)p{b\c) or p{ab) = p{a)p{b) . (2.66) 

The symmetry of these expressions implies that p {b\ac) — p {b\c) as well: if a is 
independent of b, then b is independent of a. 




P(a+b) 



Pib) 



Figure 2.1: Venn diagram showing P{a), P{b), P{ab) and P{a + h). 



Two statements or events ai and 02 are mutually exclusive given h if they 
cannot be true simultaneously, i.e., p{aia2\b) — 0. Notice that neither p{ai\b) 
nor p{a2\h) need vanish. For mutually exclusive events the sum rule simplifies 
to 

p{ai+a2\h)=p{ai\b)+p{a2\h). (2.67) 

The generalization to many mutually exclusive statements ai, 02, . . . , a,i (mu- 
tually exclusive given h) is immediate, 

n 

p(ai + a2 + • • • + a„|6) = E P{a^\h) ■ (2.68) 

1=1 

If one of the statements Oi, 02, . . . , a„ is necessarily true, i.e., they cover all 
possibilities, they are said to be exhaustive. Then their conjunction is necessarily 
true, ai + a2 + • • • + a„ = T, so that 

p(ai + a2H ha„|&)==l. (2.69) 

If, in addition to being exhaustive, the statements ai, 02, . . . , a„ are also mutu- 
ally exclusive then 

E P{a,) = 1 . (2.70) 

i=l 

A useful generalization involving the probabilities p{ai\b) conditional on any 
arbitrary proposition b is 

n 

EpKI&) = 1- (2.71) 

1=1 

The proof is straightforward: 

n n 

Pib) = p{bT) = J: p{ba{) = p{b) Y.P{a^\b) . (2.72) 
1=1 1=1 
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2.5.4 Marginalization 

Once we decide that it is legitimate to quantify degrees of belief by real numbers 
p the problem becomes how do we assign these numbers. The sum and product 
rules show how we should assign probabilities to some statements once proba- 
bilities have been assigned to others. Here is an important example of how this 
works. 

We want to assign a probability to a particular statement b. Let ai, a2, . . . , a„ 
be mutually exclusive and exhaustive statements and suppose that the proba- 
bilities of the conjunctions buj are known. We want to calculate p{b) given the 
joint probabilities p{baj). The solution is straightforward: sum pihaj) over all 



ajS, use the product rule, and eq.(2.71l to get 



= PiP)Y.P{aj\b) = p{b) . (2.73) 

3 3 

This procedure, called marginalization, is quite useful when we want to eliminate 
uninteresting variables a so we can concentrate on those variables b that really 
matter to us. The distribution p{b) is referred to as the marginal of the joint 
distribution p{ab). 

For a second use of formulas such as these suppose that we happen to know 
the conditional probabilities p{b\a). When a is known we can make good infer- 
ences about b, but what can we tell about b when we are uncertain about the 
actual value of a? Then we proceed as follows. Use of the sum and product 
rules gives 

p{b) - = Ep{b\aM^3) ■ (2-74) 

j j 

This is quite reasonable: the probability of b is the probability we would assign 
if the value of a were precisely known, averaged over all as. The assignment p(b) 
clearly depends on how uncertain we are about the value of a. In the extreme 
case when we are totally certain that a takes the particular value ak we have 
p{aj) = Sjk and we recover p{b) — p{b\ak) as expected. 



2.6 The expected value 

Suppose we know that a quantity x can take values Xi with probabilities pi. 
Sometimes we need an estimate for the quantity x. What should we choose? It 
seems reasonable that those values Xi that have larger pi should have a dominant 
contribution to x. We therefore make the following reasonable choice: The 
expected value of the quantity x is denoted by (x) and is given by 

(x)l^'Eft^. ■ (2.75) 

i 

The term 'expected' value is not always an appropriate one because (x) may 
not be one of the actually allowed values Xi and, therefore, it is not a value we 
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would expect. The expected value of a die toss is (IH h 6)/6 = 3.5 which is 

not an allowed result. 

Using the average (a;) as an estimate for the expected value of x is reason- 
able, but it is also somewhat arbitrary. Alternative estimates arc possible; for 
example, one could have chosen the value for which the probability is maximum 
- this is called the 'mode'. This raises two questions. 

The first question is whether (x) is a good estimate. If the probability distri- 
bution is sharply peaked all the values of x that have appreciable probabilities 
are close to each other and to (x). Then (x) is a good estimate. But if the 
distribution is broad the actual value of x may deviate from (x) considerably. 
To describe quantitatively how large this deviation might be we need to describe 
how broad the probability distribution is. 

A convenient measure of the width of the distribution is the root mean square 
(rms) deviation defined by 

Ax=(^{x-{x)fy^^. (2.76) 

The quantity Ace is also called the standard deviation, its square (Aa;)^ is called 
the variance. For historical reasons it is common to refer to the 'variance of a;' 
but this is misleading because it suggests that x itself could vary; Aa; refers to 
our knowledge about x. 

If Aa; <^ (x) then x will not deviate much from (a;) and we expect (a;) to be 
a good estimate. 

The definition of Aa; is somewhat arbitrary. It is dictated both by common 
sense and by convenience. Alternatively we could have chosen to define the 
width of the distribution as (|a; — (a;)|) or ((a; — (x))^)^/'' but these definitions 
are less convenient for calculations. 

Now that we have a way of deciding whether (x) is a good estimate for x 
we may raise a second question: Is there such a thing as the "best" estimate 
for X? Consider another estimate x' . We expect x' to be accurate provided the 
deviations from it are small, i.e., ((a; — x')^) is small. The best x' is that for 
which its variance is a minimum 

= 0, (2.77) 

best 

which implies x'best = (x)- Conclusion: (x) is the best estimate for x when 
by "best" we mean the one with the smallest variance. But other choices are 
possible, for example, had we actually decided to minimize the width (|x — x'|) 
the best estimate would have been the median, x'best = Xm, a value such that 
Prob(x < Xm) = Prob(x > x„i) = 1/2. 

We conclude this section by mentioning two important identities that will 
be repeatedly used in what follows. The first is that the average deviation from 
the mean vanishes. 



(x-(x)) = 0. 



(2.78) 
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because deviations from the mean are just as likely to be positive and negative. 
The second useful identity is 



(2.79) 



The proofs are trivial - just use the definition (2.751. 



2.7 The binomial distribution 

Suppose the probability of a certain event a is p. The probability of a not 
happening is 1 — p. Using the theorems discussed earlier we can obtain the 
probability that a happens m times in N independent trials. The probability 
that a happens in the first m trials and not-a or a' happens in the subsequent 
N ~ m trials is, using the product rule for independent events, p™(l — p)^"™. 
But this is only one particular ordering of the m as and the N ~ m a's. There 
are 

TV! /iV\ 

= ( I (2 80) 

ml{N — m)! \m J 

such orderings. Therefore, using the sum rule for mutually exclusive events, the 
probability of m as in TV independent trials irrespective of the particular order 
of as and as is 

P(m|TV,p) (^^^p™(l -p)^-™. (2.81) 

This is called the binomial distribution. 

Using the binomial theorem (hence the name of the distribution) one can 
show these probabilities are correctly normalized: 

^ ^ /N\ 

^F(m|TV,p)= ^ [Jp'^{l~p)^-"^ = {p+{l-p)f = l. (2.82) 



m— m— 



The range of applicability of this distribution is enormous. Whenever trials are 
independent of each other (i.e., the outcome of one trial has no infiuence on the 
outcome of another, or alternatively, knowing the outcome of one trial provides 
us with no information about the possible outcomes of another) the distribution 
is binomial. Independence is the crucial feature. 
The expected number of as is 

^ ^ /N\ 

(m) = ^ mP(m|iV,p) = ^ m jp'"(l - p)^"™. 

m=0 m=0 

This sum over m is complicated. The following elegant trick is useful. Consider 
the sum 

N 



S{p,q)=Y.mrjp"^q 



m ^N — m 
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where p and q axe independent variables. After we calculate S we will replace q 
by 1 — p to obtain the desired result, (m) = S{p, 1 — p). The calculation of S is 
easy if we note that mp™ =p §^P^- Then, using the binomial theorem 

Replacing g by 1 — p we obtain our best estimate for the expected number of as 

{m) = Np. (2.83) 



This is the best estimate, but how good is it? To answer we need to calculate 
Am. The variance is 



{Amf = ({m-{m)f) = {m-')-{mf, 



which requires we calculate (m^) 

N N 



{m''}=^m^P{m\N,p)=^m^i 

m=0 m=0 V^/ 

We can use the same trick we used before to get (m): 



m=0 

Therefore, 



(m^) = {Npf + Np{l - p), (2.84) 



and the final result for the rms deviation Am is 

Am = y/Np{l-p). (2.85) 



Now we can address the question of how good an estimate (m) is. Notice that 
Am grows with N. This might scorn to suggest that our estimate of m gets 
worse for large N but this is not quite true because (m) also grows with A''. The 
ratio 



(m) V 



shows that while both the estimate (m) and its uncertainty Am grow with N, 
the relative uncertainty decreases. 
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2.8 Probability vs. frequency: the law of large 
numbers 

Notice that the "frequency" / = m/N of as obtained in one iV-trial sequence is 
not equal to p. For one given fixed value of p, the frequency / can take any one 
of the values 2/iV, . . . N/N. What is equal to p is not the frequency 

itself but its expected value. Using eq. (2.831, 

(/>-(^>=P- (2.87) 

For large N the distribution is quite narrow and the probability that the 
observed frequency of as differs from p tends to zero as iV ^ oo. Using eq.( 2.85 1 , 



/m^ Am ^ pjl-p) 1 

■' \nJ n \I n 7V1/2 ^ 

The same ideas are more precisely conveyed by a theorem due to Bernoulli 
known as the 'weak law of large numbers'. A simple proof of the theorem 
involves an inequality due to Tchebyshev. Let p (x) dx be the probability that 
a variable X lies in the range between x and x + dx, 

P (x < X < X + dx) ~ p {x) dx. 

The variance of X satisfies 

(Ax)^ = / {x — {x))^ p [x) dx > / {x ~ (x))'^ p {x) dx , 

J J\x-{x)\>e 

where e is an arbitrary constant. Replacing {x — (x))^ by its least value gives 
{Axf>e^ [ p{x) dx = e^ P{\x~{x)\>e), 

J\x-{x)\>e 

which is Tchebyshev 's inequality, 

Ax^ ^ 



P{\x~{x)\>e)<i^—j . (2.89) 

Next we prove Bernoulli's theorem, the weak law of large numbers. First 
a special case. Let p be the probability of outcome a in an experiment E, 
P {a\E) = p. In a sequence of N independent repetitions of E the probability 
of m outcomes a is binomial. Substituting 

if)=p and (Aff^P-i^ 

into Tchebyshev's inequality we get Bernoulli's theorem, 

P{\f-p\>s\E^)<P-^l^. (2.90) 
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Therefore, the probability that the observed frequency / is appreciably different 
from p tends to zero as — > oo. Or equivalently: for any small e, the probability 
that the observed frequency / — m/N lies in the interval between p — e/2 and 
p + s/2 tends to unity as N ^ oo. 

In the mathematical/statistical literature this result is commonly stated in 
the form 

/ — > p in probability. (2-91) 

The qualifying words 'in probability' are crucial: we are not saying that the 
observed / tends to p for large N. What vanishes for large N is not the difference 
f — p itself, but rather the probability that \ f — p\ is larger than a certain (small) 
amount. 

Thus, probabilities and frequencies are not the same thing but they are 
related to each other. Since (/) — p, one might perhaps be tempted to define 
the probability p in terms of the expected frequency (/), but this does not work 
either. The problem is that the notion of expected value already presupposes 
that the concept of probability has been defined previously. The definition of a 
probability in terms of expected values is unsatisfactory because it is circular. 

The law of large numbers is easily generalized beyond the binomial distribu- 
tion. Consider the average 

1 ^ 

^=TFE2^r. (2-92) 
where xi, . . . , xn are N independent variables with the same mean (xr) = p and 



variance var(xr) — (Aa;^) — a . (In the previous discussion leading to eq.(2.90 1 
each variable Xr is either 1 or according to whether outcome a happens or not 
in the rth repetition of experiment E.) 



To apply Tchebyshev's inequality, eq.(2.89), we need the mean and the vari- 
ance of X. Clearly, 

1 ^ 1 

(^> = ^ E M = ^iV^ = . (2.93) 

Furthermore, since the independent, their variances are additive. For 
example, 

var(a;i -|- X2) — var(a;i) -|- var(x2) . (2.94) 

(Prove it.) Therefore, 

var(.)=Evar(|).iv(^)^ = ^. (2.95) 
Tchebyshev's inequality now gives, 

P{\x~f,\>e\E'')<^ (2.96) 

so that for any e > 

^lim P {\x ~ ^i\ > e\E^) ^ or ^lim P {\x - < e\E^) ^ 1 , (2.97) 
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or 

X > fj, in probability. (2.98) 

Again, what vanishes for large N is not the difference x — jjL itself, but rather 
the probability that \x — n\ is larger than any given small amount. 

2.9 The Gaussian distribution 

The Gaussian distribution is quite remarkable, it applies to a wide variety of 

problems such as the distribution of errors affecting experimental data, the 
distribution of velocities of molecules in gases and liquids, the distribution of 
fluctuations of thermodynamical quantities, and so on and on. One suspects 
that a deeply fundamental reason must exist for its wide applicability. Somehow 
the Gaussian distribution manages to codify the information that happens to 
be relevant for prediction in a wide variety of problems. The Gentral Limit 
Theorem discussed below provides an explanation. 

2.9.1 The de Moivre-Laplace theorem 

The Gaussian distribution turns out to be a special case of the binomial distri- 
bution. It applies to situations when the number N of trials and the expected 
number of as, (m) = Np, are both very large (i.e., iV large, p not too small). 

To find an analytical expression for the Gaussian distribution we note that 
when N is large the binomial distribution, 

P{m\N,p) = _^^p™(l -p)^-™, 
m\{N — my. 

is very sharply peaked: P{m\N,p) is essentially zero unless m is very close to 
(m) = Np. This suggests that to find a good approximation for P we need to 
pay special attention to a very small range of m and this can be done following 
the usual approach of a Taylor expansion. A problem is immediately apparent: 
if a small change in m produces a small change in P then we only need to keep 
the first few terms, but in our case P is a very sharp function. To reproduce 
this kind of behavior we need a huge number of terms in the series expansion 
which is impractical. Having diagnosed the problem one can easily find a cure: 
instead of finding a Taylor expansion for the rapidly varying P, one finds an 
expansion for log P which varies much more smoothly. 

Let us therefore expand log P about its maximum at mo, the location of 
which is at this point still unknown. The first few terms are 

where 

log P = log iV! — log m! — log {N — m)\ + m logp + (iV — m) log {1 — p) . 



(m - mo) + 



1 d^logP 



dm? 



(m - mo) + . . . , 



2.9. THE GAUSSIAN DISTRIBUTION 



29 



What is a derivative with respect to an integer? For large m the function log m! 
varies so slowly (relative to the huge value of log ml itself) that we may consider 
m to be a continuous variable. Then 



d log m! log m! — log (m — 1) ! , m! 
~ ; = log 



dm 



(m-1)! 



logm . 



(2.99) 



Integrating one obtains a very useful approximation - called the Stirling ap- 
proximation - for the logarithm of a large factorial 



logm 



! Ri / log a; 
Jo 



dx = (a; log x — x)\^ = mlog m — m. 



A somewhat better expression which includes the next term in the Stirling ex- 
pansion is 



log m! » TO log TO — TO -|- - log 27rTO -|- . . . 



(2.100) 



Notice that the third term is much smaller than the first two; the first two 
terms are of order m while the last is of order log to. For to = 10^"^, log to is 
only 55.3. 

The derivatives in the Taylor expansion are 



dlogP 
dm 



■ log TO -|- log {n — m) + logp — log {1 — p) = log 



p{N — to) 
to(1 — p) ' 



and 



rfMogP 
dm^ 



-N 



TO N — m m{N — to) 



To find the value mo where P is maximum set dlogP/dm ~ 0. This gives 
Too = Np = (m), and substituting into the second derivative of logP we get 



d^logP 



dm?' 



{m) 



Np{l-p) (Am)' 



Therefore 



or 



logP = logP ((m)) 



P(m) = P{{m)) exp 



(to - (m)) ^ 
2 {Amf 

(to — (m))^ 
2 {Amf 



The remaining unknown constant P((m)) can be evaluated by requiring that 
the distribution P(to) be properly normalized, that is 



1 



^ P(to) « / P{x) dx^ P{x) dx. 

m=0 "^0 
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Using 



we get 



r 

J — c 



e-"" dx = 



P((m)) = 



1 



27r {Amf 



Thus, the expression for the Gaussian distribution with mean (m) and rms 
deviation Am is 



P(m) 



-^27r(Am)^ 



exp 



(m — {m)y 
2 (Am)^ 



(2.101) 



It can be rewritten as a probabiUty for the frequency / = m/N using (m) = Np 



and (Am) = A/'p(l -p). 
1/N is 



The probabiUty that / Ues in the small range df = 



Pif)df = 



exp 



2< 



(2.102) 



where cr^ 



p(l-p)/iV. 

To appreciate the significance of the theorem consider a macroscopic variable 

X built up by adding a largo number of small contributions, x = X^^^Li Cn, where 
the are statistically independent. We assume that each ^„ takes the value e 
with probability p, and the value with probability 1 — p. Then the probability 
that X takes the value me is given by the binomial distribution P(m\N,p). For 
large N the probability that x lies in the small range me ±dx/2 where dx = e 
is 



p{x)dx 



y^27r(Aa;)' 



exp 



(^-(^)r 

2 {Axf 



dx 



(2.103) 



where (x) = Npe and (Ax)^ = Np{l — p)e'^. Thus, the Gaussian distribution 
arises whenever we have a quantity that is the result of adding a large number 
of small independent contributions. The derivation above assumes that the 
microscopic contributions are discrete (binomial, either or e), and identically 
distributed but, as shown in the next section, both of these conditions can be 
relaxed. 



2.9.2 The Central Limit Theorem 

Consider the average 



1 ^ 

x=^r^Xr, (2.104) 



2. 9. THE GA USSIAN DISTRIB UTION 



31 



of N independent variables xi, . . . , xat. Our goal is to calculate the probability 
of X in the limit of large N. Let Pr{xr) be the probability distribution for the 
rth variable with 

(2.105) 



/ir and (Aa;^)'^ = cr^ 



The probability density for x is given by the integral 

/ 1 ^ \ 

P{x) ^ Jdxi...dxN Pi{xi) ...pn{xn)S [x - —J2^r] ■ (2.106) 

(This is just an exercise in the sum and product rules.) To calculate P (x) 
introduce the averages 



def 1 ^ , _2 dcf 1 ^ 



M = T7 E Mr and a 

1\ r=l 



1 V — \ 



(2.107) 



and consider the distribution for the variable x ~ p, which is Pr(a; — /i) = P{x). 
It's Fourier transform, 



F{k) = / dx Vr{x - /i)e*'=(^-'') = j dx P{x)e'^^''-f'^ 
dxi . . . dxM Pi{xi) . ■ .pn{xn) exp 
can be rearranged into a product 



ik ^ 

-jTf X/ i-^r ~ Mr) 



(ixipi(xi)e*« 



dXNPN{xN)e''« 



(2.108) 



The Fourier transform f(k) of a distribution p(^) has many interesting and 
useful properties. For example, 



/(fc) = / d^piOe'"^ - (e^'^^) 



and the series expansion of the exponential gives 



\„=o n\ I „=o 



(*fc)" 



(2.109) 



(2.110) 



In words, the coefficients of the Taylor expansion of /(fc) give all the moments 
of p(^). The Fourier transform /(fc) is called the moment generating function 
and also the characteristic function of the distribution. 



Going back to our calculation of P{x), eq.(2.106), its Fourier transform, 
eq.( |2.108| ) is, 

N i, 

m = n fri^) , (2.111) 

r=l 
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where 



k I , \2\ 

1 + (a:,. - Mr) - ^ {(Xr - fJ-r) ) + 



-^-2# + nM) • (2.112) 



For a sufficiently large TV this can be written as 



frd) ( -'^^ ■ (2.113) 



so that 

m = =p(-^|.?)=„p(-^) . ,2.114, 

Finally, taking the inverse Fourier transform, we obtain the desired result, which 
is called the central limit theorem 

"■■<^-'"" = ^« = 7iSw"K'w) ■ 

To conclude we comment on its significance. We have shown that almost 
independently of the form of the distributions Pr {xr) the distribution of the 
average x is Gaussian centered at p, with standard deviation jN . Not only 
the Pr (a^r) need not be binomial, they do not even have to be equal to each other. 
This helps to explain the widespread applicability of the Gaussian distribution: 
it applies to almost any 'macro-variable' (such as a;) that results from adding a 
large number of independent 'micro- variables' (such as Xr/N). 

But there are restrictions; although very common, Gaussian distributions do 
not obtain always. A careful look at the derivation above shows the crucial step 



was taken in eqs.(2.112l and (2.1141 where we neglected the contributions of the 
third and higher moments. Earlier we mentioned that the success of Gaussian 
distributions is due to the fact that they codify the information that happens to 
be relevant to the particular phenomenon under consideration. Now we see what 
that relevant information might be: it is contained in the first two moments, 
the mean and the variance - Gaussian distributions are successful when third 
and higher moments are irrelevant. (This can be stated more precisely in terms 
as the so-called Lyapunov condition.) 

Later we shall approach this same problem from the point of view of the 
method of maximum entropy and there we will show that, indeed, the Gaussian 
distribution can be derived as the distribution that codifies information about 
the mean and the variance while remaining maximally ignorant about everything 
else. 
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2.10 Updating probabilities: Bayes' rule 

Now that we have solved the problem of how to represent a state of knowledge 
as a consistent web of interconnected beliefs we can address the problem of 
updating from one consistent web of beliefs to another when new information 
becomes available. We will only consider those special situations where the 
information to be processed is in the form of data. 

Specifically the problem is to update our behefs about 6 (either a single 
parameter or many) on the basis of data x (either a single number or several) 
and of a known relation between and x. The updating consists of replacing the 
prior probability distribution p{0) that represents our beliefs before the data is 
processed, by a posterior distribution Pnew(^) that applies after the data has 
been processed. 

2.10.1 Formulating the problem 

We must first describe the state of our knowledge before the data has been 
collected or, if the data has already been collected, before we have taken it into 
account. At this stage of the game not only we do not know 9, we do not know 
X either. As mentioned above, in order to infer 9 from x we must also know how 
these two quantities are related to each other. Without this information one 
cannot proceed further. Fortunately we usually know enough about the physics 
of an experiment that if 9 were known we would have a fairly good idea of what 
values of x to expect. For example, given a value 9 for the charge of the electron, 
we can calculate the velocity x of an oil drop in Millikan's experiment, add 
some uncertainty in the form of Gaussian noise and we have a very reasonable 
estimate of the conditional distribution p{x\9). The distribution p{x\6) is called 
the sampling distribution and also (less appropriately) the likelihood. We will 
assume it is known. 

We should emphasize that the crucial information about how x is related to 
is contained in the functional form of the distribution p{x\9) - say, whether 
it is a Gaussian or a Cauchy distribution - and not in the actual values of the 
arguments x and 9 which are, at this point, still unknown. 

Thus, to describe the web of prior beliefs wc must know the prior p(9) and 
also the sampling distribution p{x\0). This means that we must know the full 
joint distribution, 

p{9,x) =p{9)p{x\9) . (2.116) 

This is very important: we must be clear about what we are talking about. The 
relevant universe of discourse is neither the space 6 of possible parameters 9 nor 
is it the space X of possible data x. It is rather the product space x X and 
the probability distributions that concern us are the joint distributions p(0,x). 

Next we collect data: the observed value turns out to be X. Our goal is 
to use this information to update to a web of posterior beliefs represented by 
a new joint distribution pncwi0, x). How shall we choose ]3,icw(^, a^)? The new 
data tells us that the value of x is now known to be X. Therefore, the new web 
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of beliefs must be such that 



Pncw(x) = / d0Pncw(0, x) = S{x - X) . (2.117) 

(For simplicity we have here assumed that x is a continuous variable; had x been 
discrete Dirac Ss would be replaced by Kronecker i5s.) This is all we know but 
it is not sufficient to determine Pncw{Q, x). Apart from the general requirement 
that the new web of beliefs must be internally consistent there is nothing in any 
of our previous considerations that induces us to prefer one consistent web over 
another. A new principle is needed. 



2.10.2 Minimal updating: Bayes' rule 

The basic updating strategy that we adopt below reflects the conviction that 
what we have learned in the past, the prior knowledge, is a valuable resource 
that should not be squandered. Prior beliefs should be revised only when this is 
demanded by the new information; the new web of beliefs should coincide with 
the old one as much as possible. We propose to adopt the following 

Principle of Minimal Updating (PMU): The web of beliefs needs to be 
revised only to the extent required by the new data. 

This seems so reasonable and natural that an explicit statement may seem 
superfluous. The important point, however, is that it is not logically necessary. 
We could update in many other ways that preserve both internal consistency 
and consistency with the new information. 



As we saw above the new data, eq.(2.117 1, does not fully determine the joint 
distribution 



Pncw{0, x) = Pncw{x)Pncw{d\x) = S{x - X)p^cw{S\x) . (2.118) 

All distributions of the form 

P„ew(^, x) = S{x - X)p^,^{9\X) , (2.119) 

where PncwiO\X) remains arbitrary is compatible with the newly acquired data. 
We still need to assign pngyj{9\X). It is at this point that we invoke the PMU. 
We stipulate that no further revision is needed and set 

PnUW = Pom{0\X) = pi0\X) . (2.120) 

Therefore, the web of posterior beliefs is described by 

Pnow(^, x) = Six - X)pie\X) . (2.121) 

The posterior probability Pncw(^) is 

Pnew(^) = / dxpne^ie, x) ^ J dx 6{x - X)pi9\X) , (2.122) 
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or, 

Pne^iO) = p{0\X) . (2.123) 

In words, the posterior probability equals the prior conditional probability of 
given X. This result, known as Bayes' rule, is extremely reasonable: we 
maintain those beliefs about 6 that are consistent with the data values X that 
turned out to be true. Data values that were not observed are discarded because 
they are now known to be false. 'Maintain' is the key word: it reflects the PMU 
in action. 

Using the product rule 



p{9, X) = p{e)p{x\e) = p{x)p{e\x) , (2.124) 

Bayes' rule can be written as 

PncAd) = P{0)^^ . (2.125) 
Remark: Bayes' rule is usually written in the form 

and called Bayes' theorem. This formula is very simple; perhaps it is too simple. 



It is just a restatement of the product rule, eq.(2.124l, and therefore it is a 
simple consequence of the internal consistency of the prior web of beliefs. The 
drawback of this formula is that the left hand side is not the posterior but 
rather the prior conditional probability; it obscures the fact that an additional 
principle - the PMU - was needed for updating. 



The interpretation of Bayes' rule is straightforward: according to eq.(2.125l 
the posterior distribution Pncw{S) gives preference to those values of 6 that were 
previously preferred as described by the prior p{6), but this is now modulated 
by the likelihood factor p(X\9) in such a way as to enhance our preference for 
values of 6 that make the data more likely, less surprising. The factor in the 
denominator p{X) which is the prior probability of the data is given by 



piX)^Jpi9)piX\9)de, 



(2.127) 



and plays the role of a normalization constant for the posterior distribution 
Pnow(^)- It does not help to discriminate one value of 9 from another because it 
affects all values of 9 equally and is therefore not important except, as we shall 
see later in this chapter, in problems of model selection. 

Neither the rule, eq.(2.123), nor the theorem, eq.(2.126l, was ever actually 
written down by Bayes. The person who first explicitly stated the theorem and, 
more importantly, who first realized its deep significance was Laplace. 
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ExEimple: is there life on Metrs? 

Suppose wc arc interested in whether there is hfe on Mars or not. How is 
the probability that there is hfe on Mars altered by new data indicating the 
presence of water on Mars. Let 9 = 'There is life on Mars'. The prior information 
includes the fact / = 'All known life forms require water'. The new data is that 
X = 'There is water on Mars'. Let us look at Bayes' rule. We can't say much 
about p{X\I) but whatever its value it is definitely less than 1. On the other 
hand p{X\9I) w 1. Therefore the factor multiplying the prior is larger than 
1. Our belief in the truth of 9 is strengthened by the new data X. This is 
just common sense, but notice that this kind of probabilistic reasoning cannot 
be carried out if one adheres to a strictly frequentist interpretation there is 
no set of trials. The name 'Bayesian probabilities' given to 'degrees of belief 
originates in the fact that it is only under this interpretation that the full power 
of Bayes' rule can be exploited. 



Example: testing positive for a rare disease 

Suppose you are tested for a disease, say cancer, and the test turns out to be 
positive. Suppose further that the test is said to be 99% accurate. Should you 
panic? It may be wise to proceed with caution. 

One should start by explaining that '99% accurate' means that when the test 
is applied to people known to have cancer the result is positive 99% of the time, 
and when applied to people known to be healthy, the result is negative 99% of 
the time. We express this accuracy as p{y\c) = A = 0.99 and p{n\c) = A = 0.99 
{y and n stand for 'positive' and 'negative', c and c stand for 'cancer' or 'no 
cancer'). There is a 1% probability of false positives, p{y\c) = 1 — A, and a 1% 
probability of false negatives, p{n\c) = 1 — A. 

On the other hand, what we really want to know is Pnew(c) = p{c\y), the 
probability of having cancer given that you tested positive. This is not the 
same as the probability of testing positive given that you have cancer, p{y\c); 
the two probabilities are not the same thing! So there might be some hope. The 
connection between what we want, p{c\y), and what we know, p{y\c), is given 
by Bayes' theorem. 

An important virtue of Bayes' rule is that it doesn't just tell you how to 
process information; it also tells you what information you should seek. In this 
case one should find p{c), the probability of having cancer irrespective of being 
tested positive or negative. Suppose you inquire and find that the incidence of 
cancer in the general population is 1%; this means that p{c) = 0.01. Thus, 

One also needs to know p{y), the probability of the test being positive irre- 
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spective of whether the person has cancer or not. To obtain p{y) use 

v{y) pyy) 

and p{c\y) + p{c\y) = 1 which leads to our final answer 

^M'"' p(cM + (l''-pt))(l-.4) ' 

For an accuracy A = 0.99 and an incidence p{c) = 0.01 we get p{c\y) = 50% 
which is not nearly as bad as one might have originally feared. Should one 
dismiss the information provided by the test as misleading? No. Note that the 
probability of having cancer prior to the test was 1% and on learning the test 
result this was raised all the way up to 50%. Note also that when the disease 
is really rare, p{c) 0, wc still get p{c\y) even when the test is quite 
accurate. This means that for rare diseases most positive tests turn out to be 
false positives. 

Wc conclude that both the prior and the data contain important information; 
neither should be neglected. 

Remark: The previous discussion illustrates a mistake that is common in verbal 

discussions: if h denotes a hypothesis and e is some evidence, it is quite obvious 
that we should not confuse p{e\h) withp(/i|e). However, when expressed verbally 
the distinction is not nearly as obvious. For example, in a criminal trial jurors 
might be told that if the defendant were guilty (the hypothesis) the probability 
of some observed evidence would be large, and the jurors might easily be misled 
into concluding that given the evidence the probability is high that the defendant 
is guilty. Lawyers call this the prosecutor's fallacy. 



Example: uncertain data 

As before we want to update from a prior joint distribution p{9, x) = p{x)p{9\x) 
to a posterior joint distribution Pncvi{0, x) ~ Pncvi{x)pncw{d\x) when information 
becomes available. When the information is data X that precisely fixes the value 
of X, we impose that Pnew(a;) = 5{x — X). The remaining unknown p^Q^{9\x) 
is determined by invoking the PMU: no further updating is needed. This fixes 
Pnew(^'|.7') to be the old p{9\x) and yields Bayes' rule. 

It may happen, however, that there is a measurement error. The data X 
that was actually observed does not cxHistrain the value of x c;ompletcly. To be 
explicit let us assume that the remaining uncertainty in x is well understood: 
the observation X constrains our beliefs about a; to a distribution Px{x) that 
happens to be known. Px(.r) could, for example, be a Gaussian distribution 
centered at X, with some known standard deviation a. 

This information is incorporated into the posterior distribution, Pnew(^, x) = 
Pncw{x)Pacw{d\x) , by imposing that pac^^■{x) = Px{x). The remaining condi- 
tional distribution is, as before, determined by invoking the PMU, 



Pney/{e\x) =Po\d{0\x) =p{0\x) , 



(2.129) 
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and therefore, the joint posterior is 

Pne^{e,x) = Px{x)pi9\x) . (2.130) 
Marginalizing over the uncertain x yields the new posterior for 9, 

Pnew(e) = JdxPxix)p{0\x) . (2.131) 

This generalization of Bayes' rule is sometimes called Jeffrey's conditionalization 
rule. 

Incidentally, this is an example of updating that shows that it is not always 
the case that information comes purely in the form of data X. In the derivation 
above there clearly is some information in the observed value X and some in- 
formation in the particular functional form of the distribution Px{x), whether 
it is a Gaussian or some other distribution. 

The common element in our previous derivation of Bayes' rule and in the 
present derivation of Jeffrey's rule is that in both cases the information being 
processed is a constraint on the allowed posterior marginal distributions Pncw{x) . 
Later we shall see (chapter 5) how the updating rules can be generalized still 
further to apply to even more general constraints. 



2.10.3 Multiple experiments, sequential updating 

The problem here is to update our beliefs about 9 on the basis of data xi,X2, ■ ■ ■ ,Xr, 
obtained in a sequence of experiments. The relations between 9 and the vari- 
ables Xi are given through known sampling distributions. We will assume that 
the experiments are independent but they need not be identical. When the ex- 
periments are not independent it is more appropriate to refer to them as being 
performed is a single more complex experiment the outcome of which is a set of 
numbers {xi, . . . , a;„}. 

For simplicity we deal with just two identical experiments. The prior web 
of beliefs is described by the joint distribution, 

p{xi,X2,0) - Pi9) p{xi\e)p{x2\9) = p{xi)p{9\xi)p{x2\e) , (2.132) 

where we have used independence, p{x2\9,xi) ~ p(x2\9\ 

The first experiment yields the data x\ = Xi. Bayes' rule gives the updated 
distribution for as 

p,{9)=p{9\X,)^p{9)P^^ . (2.133) 
The second experiment yields the data X2 = X2 and requires a second applica- 



tion of Bayes' rule. The posterior pi{9) in eq.(2.133l now plays the role of the 
prior and the new posterior distribution for 6 is 

pM9) = Pi{0\X2) = Pi{0)^^^^ ■ (2.134) 
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We have explicitly followed the update from p{9) to pi{9) to pi2{9). It is 
straightforward to show that the same result is obtained if the data from both 
experiments were processed simultaneously, 



piXi,X2\9) 

PiXi,X2) 



(2.135) 



Indeed, using eq.(2.132l and ( 2.133[ ), this last equation can be rewritten as 



p,2{9)=pi9) 



p{Xi\9) p{X2\9) 
p{X,) p{X2\X,) 



pm 



p{X2\9) 

p{X2\Xi) 



(2.136) 



and it remains to show that p(X2\Xi) 
ward; use eq.(2.134l and (2.133): 



pi{X2). This last step is straightfor- 



Pi{X2) = Jpi{9)p{X2\9)d9 

rP{Xl,X2, 



Ipiof-^p{X2\e)d9 



PiXi) 



p(.x,) 

d9 = piX2\Xi) . 



(2.137) 



From the symmetry of eq.(2.135) it is clear that the same posterior pi2{9) 



is obtained irrespective of the order that the data Xi and X2 are processed. 
The commutativity of Bayesian updating follows from the special circumstance 
that the information conveyed by one experiment does not revise or render 
obsolete the information conveyed by the other experiment. As we generalize our 
methods of inference for processing other kinds of information that do interfere 
with each other (and therefore one may render the other obsolete) we should 
not expect, much less demand, that commutativity will continue to hold. 



2.10.4 Remarks on priors 

Let us return to the question of the extent to which probabilities incorporate 
subjective and objective elements. We have seen that Bayes' rule allows us 
to update from prior to posterior distributions. The posterior distributions 
incorporate the presumably objective information contained in the data plus 
whatever earlier beliefs had been codified into the prior. To the extent that the 
Bayes updating rule is itself unique one can claim that the posterior is "more 
objective" than the prior. As we update more and more we should expect that 
our probabilities should reflect more and more the input data and less and less 
the original subjective prior distribution. In other words, some subjectivity 
is unavoidable at the beginning of an inference chain, but it can be gradually 
suppressed as more and more information is processed. 

The problem of choosing the first prior in the inference chain is a difficult 
one. We will tackle it in several different ways. Later in this chapter, as we 
introduce some elementary notions of data analysis, we will address it in the 
standard way: just make a "reasonable" guess ~ whatever that might mean. 
With experience and intuition this seems to work well. But when addressing 
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new problems we have neither experience nor intuition and guessing is risky. We 
would like to develop more systematic ways to proceed. Indeed it can be shown 
that certain types of prior information (for example, symmetries and/or other 
constraints) can be objectively translated into a prior once we have developed 
the appropriate tools - entropy and geometry. (See e.g. [Caticha Preuss 04] 
and references therein.) 

Our immediate goal here is, first, to remark on the dangerous consequences of 
extreme degrees of belief, and then to prove our previous intuitive assertion that 
the accumulation of data will swamp the original prior and render it irrelevant. 

Dangerous extremes: the prejudiced mind 

The consistency of Bayes' rule can be checked for the extreme cases of certainty 
and impossibility: Let B describe any background information. li p{9\B) = 1, 
then eB = B and p{X\eB) = p{X\B), so that Bayes' rule gives 



A similar argument can be carried through in the case of impossibility; If 
p{9\B) = 0, then Pncvi{()\B) = 0. Conclusion: if we are absolutely certain 
about the truth of 9, acquiring data X will have absolutely no effect on our 
opinions; the new data is worthless. 

This should serve as a warning to the dangers of erroneously assigning a 
probability of 1 or of 0: since no amount of data could sway us from OTir prior 
beliefs we may decide we did not need to collect the data in the first place. If 
you are absolutely sure that Jupiter has no moons, you may either decide that 
it is not necessary to look through the telescope, or, if you do look and you 
see some little bright spots, you will probably decide the spots are mere optical 
illusions. Extreme degrees of behef are dangerous: a truly prejudiced mind does 
not, and indeed, cannot question its own beliefs. 

Lots of data overwhelms the prior 

As more and more data is accumulated according to the sequential updating 
described earlier one would expect that the continuous inflow of information 
will eventually render irrelevant whatever prior information we might have had 
at the start. This is indeed the case: unless we have assigned a pathological 
prior - all we need is a prior that is smooth where the likelihood is large - after 
a large number of experiments the posterior becomes essentially independent of 
the prior. 

Consider N independent repetitions of a certain experiment E that yield the 
data X = {Xi . . . X^}- The corresponding likelihood is 



p^,^{9\B) = p{9\B) 



P{X\9B) 
p{X\B) 



= 1 . 



(2.138) 



N 



p{x\9)= Y{p{Xr\e) , 



(2.139) 



r=l 
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and the posterior distribution j'now(^) is 

To investigate the extent to which the data X supports the particular value 
01 rather than any other value 02 it is convenient to study the ratio 

where we introduce the likelihood ratios, 

RiX)"^ f{Rr{X,) and R^iX,)"^ "^-^^^^ . (2.142) 

We want to prove the following theorem: Barring two trivial exceptions, for any 
arbitrarily large positive A, we have 

Ymi P{R{X)> K\ei) = l (2.143) 

A''^oo 

or, in other words, 

given Oi, R{X) — > oo in probability. (2.144) 

The significance of the theorem is that as data accumulates a rational person 
becomes more and more convinced of the truth - in this case the true value is 
01 - and this happens essentially irrespective of the prior p(9) . 

The theorem fails in two cases: first, when the prior p{Oi) vanishes, in 
which case probabilities conditional on 9i are meaningless, and second, when 
p{Xr\9i) = p{Xr\92) for all Xr which describes an experiment E that is flawed 
because it cannot distinguish between 9i and 02- 

The proof of the theorem is an application of the weak law of large numbers. 
Consider the quantity 

^logR{X) = ^^^logRr{Xr) (2.145) 



Since the variables log i?r(-'^^r) are independent, eq.(2.97l gives 



lim P 

N^oo 



j^logRiX)~K{9,,92] 



< £\0i = 1 (2.146) 



where e is any small positive number and 

K{e,,92) = /j^logR{X)\9, 



^j:p{Xr\Oi)logRr{X,) . (2.147) 



42 



CHAPTER 2. PROBABILITY 



In other words, 

given 01, e^(^"'') < R{X) < e^(^+^) m probability. (2.148) 

In Chapter 4 we will prove that K{9i, 62) > with equality if and only if the two 
distributions p{Xr\Oi) and p{Xr\92) are identical, which is precisely the second 
of the two trivial exceptions we explicitly avoid. Thus K{9i,92) > 0, and this 
concludes the proof. 

We see here the first appearance of a quantity, 

K{e,,e2) = +Ep(^r|ei)iog4ft4 ' (2.149) 

that will prove to be central in later discussions. When multiplied by — 1, 
the quantity —K{9i,02) is called the relative entropy^ that is the entropy of 
p{Xr\Oi) relative to p{Xr\92)- It can be interpreted as a measure of the extent 
that the distribution p{Xr\Oi) can be distinguished from p{Xr\02)- 



2.11 Examples from data analysis 

To illustrate the use of Bayes' theorem as a tool to process information when 
the information is in the form of data we consider some elementary examples 
from the field of data analysis. (For detailed treatments that are friendly to 
physicists see e.g. [Sivia Skilling 06, Gregory 05].) 

2.11.1 Parameter estimation 

Suppose the probabihty for the quantity x depends on certain parameters 6, 
p = p{x\6). Although most of the discussion here can be carried out for an 
arbitrary function p it is best to be specific and focus on the important case of 
a Gaussian distribution. 

The objective is to estimate the parameters 6 = (/i, a) on the basis of a set of 
data X — {Xi, . . .Xn}. We assume the measurements are statistically inde- 
pendent of each other and use Bayes' theorem to get 

p(m, ^I^) = Upix^\^^, ^) ■ (2.151) 

Independence is important in practice because it leads to considerable practical 
simplifications but it is not essential: instead of N independent measurements 



^ Other names include relative information, directed divergence, and Kullback-Leibler dis- 
tance. 
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each providing a single datum we would have a single complex experiment that 
provides N non-independent data. 



Looking at eq.(2.151l we see that a more precise formulation of the same 



problem is the following. We want to estimate certain parameters 6, in our case 
H and tJ, from repeated measurements of the quantity x on the basis of several 
pieces of information. The most obvious is 



1. The information contained in the actual values of the collected data X. 



Almost equally obvious (at least to those who are comfortable with the Bayesian 
interpretation of probabilities) is 

2. The information about the parameters that is codified into the prior dis- 
tribution p{9). 

Where and how this prior information was obtained is not relevant at this point; 
it could have resulted from previous experiments, or from other background 
knowledge about the problem. The only relevant part is whatever ended up 
being distilled into p{9). 

The last piece of information is not always explicitly recognized; it is 

3. The information that is codified into the functional form of the 'sampling' 
distribution p{X\9). 

If we are to estimate parameters 6 on the basis of measurements of a quantity 
X it is clear that we must know how 9 and x are related to each other. Notice 
that item 3 refers to the functional form - whether the distribution is Gaussian 
as opposed to Poisson or binomial or something else - and not to the actual 
values of the data X which is what is taken into account in item 1. The nature 
of the relation in p{X\9) is in general statistical but it could also be completely 
deterministic. For example, when X is a known function of 9, say X = f{9), we 
have p{X\9) = S[X ~ f{9)]. In this latter case there is no need for Bayes' rule. 
Eq. ( 2.151[ ) is rewritten as 



p{n,a\X) 



p(Ai,cr) 



1 



exp 



(2.152) 



Introducing the sample average X and sample variance s^, 



1 



N 



N 



X^—yx, and 



eq.(2.152l becomes 



p{li,a\X) 



pin, a) 



1 



P{X) (2^a2)^/2 



exp 



{i^-xy+. 

2(7^ /N 



(2.153) 



(2.154) 
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It is interesting that the data appears here only in the particular combination 



in eq.(2.153l - different sets of data characterized by the same X and s lead to 



the same inference about /i and a. (As discussed earlier the factor p {X) is not 
relevant here since it can be absorbed into the normalization of the posterior 
p{t^,a\X).) 

Eq. ( |2.154 1 incorporates the information described in items 1 and 3 above. 
The prior distribution, item 2, remains to be specified. Let us start by consid- 
ering the simple case where the value of a is actually known. Then j3(/i, a) = 
p{fj,)S{a — (Tq) and the goal is to estimate ^. Bayes' theorem is now written as 



p{fi\X) = 



p(m) 1 



exp 



exp 



2al/N 



(2.155) 



a p(/i) exp 



X) 



2allN 



(2.156) 



Suppose further that we know nothing about /i; it could have any value. This 
state of extreme ignorance is represented by a very broad distribution that we 
take as essentially uniform within some large range; /i is just as likely to have one 
value as another. For p(/i) ~ const the posterior distribution is Gaussian, with 
mean given by the sample average x, and variance <Jq/N. The best estimate 
for the value of is the sample average and the uncertainty is the standard 
deviation. This is usually expressed in the form 



_CTO_ 



(2.157) 



Note that the estimate of /i from N measurements has a much smaller error 
than the estimate from just one measurement; the individual measurements are 
plagued with errors but they tend to cancel out in the sample average. 

In the case of very little prior information - the uniform prior - we have re- 
covered the same results as in the standard non-Bayesian data analysis approach. 
The real difference arises when prior information is available: the non-Bayesian 
approach can't deal with it and can only proceed by ignoring it. On the other 
hand, within the Bayesian approach prior information is easily taken into ac- 
count. For example, if we know on the basis of other physical considerations 
that /i has to be positive we assign p{(x) = for /i < and we calculate the 



estimate of /i from the truncated Gaussian in eq.(2.156) 



A slightly more complicated case arises when the value of a is not known. 
Let us assume again that our ignorance of both and a is quite extreme and 
choose a uniform prior. 



p(/i,cr) cx 



C for cr > 
otherwise. 



(2.158) 
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Another popular choice is a prior that is uniform in /i and in log a. When there 
is a considerable amount of data the two choices lead to practically the same 
conclusions but we see that there is an important question here: what do we 
mean by the word 'uniform'? Uniform in terms of which variable? a, or cr^, or 
log a ? Later we shall have much more to say about this misleadingly innocuous 
question. 

To estimate fj, we return to eq.(2.152l or (2.1541. For the purpose of estimat- 
ing /i the variable a is an uninteresting nuisance which, as discussed in section 
2.5.4, is eliminated through marginalization. 



p{^l\X) = Jdap{^l,a\X) 





oo 

(X Jda 



1 



(2.159) 
(2.160) 

(2.161) 
(2.162) 



which is called the Student-t distribution. Since the distribution is symmetric 
the estimate for /i is easy to get, 



■ exp 



Change variables to t = then 

oo 

p{^x\X) oc Jdtt^-^' 



Repeated integrations by parts lead to 

p(/i|X)cx \N({^I-xY + . 



X) 



(2.163) 



The posterior p(fj,\X) is a Lorentzian-likc function raised to some power. As the 
number of data grows, say N > 10, the tails of the distribution are suppressed 
and p(fi\X) approaches a Gaussian. To obtain an error bar in the estimate 
fi = X we can estimate the variance of /i using the following trick. Note that 
for the Gaussian in eq.(2.150l. 



dx'^ 



log pix\n,a) 



1 



(2.164) 



Therefore, to the extent that eq.(2.162l approximates a Gaussian, we can write 

d^ 



d^ 



2 log p{^i\x) 



N -I 



(2.165) 



(This explains the famous factor of iV — 1 . As we can see it is not a particularly 
fundamental result; it follows from approximations that are meaningful only for 
large N .) 
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Wc can also estimate a directly from the data. This requires that we 
marginalize over fi, 



p{a\X)= J dtxp{f,,a\X) 



1 

-jv exp 



2a2 



/ d/j. exp 



(2.166) 
(2.167) 



1 /2 

The Gaussian integral over /x is (2Tra'^/N) oc a and therefore 



pia\X) (X exp 

fT 



2^ 



(2.168) 



As an estimate for a we can use the value where the distribution is maximized, 



N 



N-1 



which agrees with our previous estimate of (A/u) , 



N N-1 



(2.169) 



(2.170) 



An error bar for a itself can be obtained using the previous trick (provided N 
is large enough) of taking a second derivative of log p. The result is 



Cr = CTmax ± 



V2(iV-l) ■ 



(2.171) 



2.11.2 Curve fitting 

The problem of fitting a curve to a set of data points is a problem of parameter 
estimation. There are no new issues of principle to be resolved. In practice, how- 
ever, it can be considerably more complicated than the simple cases discussed 
in the previous paragraphs. 

The problem is as follows. The observed data is in the form of pairs (Xj, Yi) 
with i = 1,. . .N and we believe that the true ys are related to the Xs through 
a function yi = fe{xi) which depends on several parameters 9. The goal is to 
estimate the parameters 6 and the complication is that the measured values of 
y are afflicted by experimental errors, 

Y^fe{Xi) + e,. (2.172) 

For simplicity we assume that the probability of the error £j is Gaussian with 
mean {si) = and that the variances (ef ) = are known and the same for all 
data pairs. We also assume that there are no errors affecting the Xs. A more 
realistic account might have to reconsider these assumptions. 
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The sampling distribution is 



piY\9) = n PiYi\e) , 

i=l 



where 

p{Yi\e) = 

Bayes' theorem gives, 



: exp - 



2a2 



p{0\Y) (xp{0)exp 



(2.173) 



(2.174) 



(2.175) 



As an example, suppose that we are trying to fit a straight line through data 
points 

f{x) = a + bx, (2.176) 

and suppose further that being ignorant about the values of 9 = (a, b) we choose 
p{6) = p{a, b) const, then 



p{a, b\Y) oc exp 



^{Yi-a-bXif 



2<t2 



(2.177) 



A good estimate of a and b is the value that maximizes the posterior distribution, 

which as we see, is equivalent to using the method of least squares. But this 
Bayesian analysis, simple as it is, can already give us more: from p(a, b\Y) we 
can also estimate the uncertainties Aa and Ab which lies beyond the scope of 
least squares. 



2.11.3 Model selection 

Suppose we are trying to fit a curve y = feix) through data points {Xi,Yi), 
i = 1, . . . iV. How do we choose the function /g? To be specific let fe be a 
polynomial of order n, 

feix) = eo + eix + ...+ 9nx'', (2.178) 

the techniques of the previous section allow us to estimate the parameters 
9q, . . . , 9n but how do wc decide the order n? Should we fit a straight or a 
quadratic line? It is not obvious. Having more parameters means that we will 
be able to achieve a closer fit to the data, which is good, but we might also be 
fitting the noise, which is bad. The same problem arises when the data shows 
peaks and we want to estimate their location, their width, and their number; 
could there be an additional peak hiding in the noise? Are we just fitting noise, 
or does the data really support one additional peak? 

We say these are 'problems of model selection'. To appreciate how important 
they can be consider replacing the modestly unassuming word 'model' by the 
more impressive sounding word 'theory'. Given two competing theories, which 
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one does the data support best? What is at stake is nothing less than the 
foundation of experimental science. 

On the basis of data X we want to select one model among several competing 
candidates labeled hy m — 1,2,... Suppose model m is defined in terms of some 
parameters 9m — {9mi,9m2, ■ ■ ■} and their relation to the data X is contained in 
the samphng distribution p{X\m,6rn)- The extent to which the data supports 
model TO, i.e., the probability of model to given the data, is given by Bayes' 
theorem, 

Pirn\X) = ^^p{X\m) , (2.179) 

where p{ni) is the prior for the model. The factor p{X\m), which is the prior 
probability for the data given the model, plays the role of a likelihood. It is often 
called the 'evidence'. This is not altogether appropriate because the meaning 
of p{X\m) is already given as "the prior probability of the data." There is 
nothing more to be said about it. Calling it the 'evidence' can only mislead 
us by suggesting interpretations and therefore uses that go beyond and could 
conceivably be in conflict with its probability meaning After this warning, we 
follow standard practice. The "evidence" is calculated from 



p{X\m) = J dOm piX,0m\m) = J d0m Pi0m\m) p(X|to,0™) . (2.180) 
Therefore 

p{m\X)ocp{m) [ dOm p{e„,\m)p{X\m,em) ■ (2.181) 



Thus, the problem is solved, at least in principle, once the priors p{m) and 
p{6m\fn) are assigned. Of course, the practical problem of calculating the multi- 
dimensional integrals can still be quite formidable. 

No further progress is possible without making specific choices for the various 



functions in eq.(2.181) but we can offer some qualitative comments. When 
comparing two models, TOi and TO2, it is fairly common to argue that a priori 
we have no reason to prefer one over the other and therefore we assign the 
same prior probability p{mi) ~ p{m2). (Of course this is not always justified. 
Particularly in the case of theories that claim to be fundamental people usually 
have very strong prior prejudices favoring one theory against the other. Be that 
as it may, let us proceed.) 

Suppose the prior p(6m\'m) represents a uniform distribution over the pa- 
rameter space. Since 

dOm p{em\m) ^ 1 then p(0„|m) « , (2.182) 

where Vm is the 'volume' of the parameter space. Suppose further that p{X\m, 9m) 
has a single peak of height imax spread out over a region of 'volume' 59m- The 



•^A similar problem occurs when {x) is called the "expected" value. It misleads us into 
thinking that {x) is the value we should expect, which is not necessarily true. 
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value 9m where p{X\m, 0„i) attains its maximum can be used as an estimate 
for 9m and the 'volume' 69m is then interpreted as an uncertainty. Then the 
integral oip{X\m,9m) can be approximated by the product Lmax x 59m- Thus, 
in a very rough and quaHtative way the probabihty for the model given the data 
is 

p{m\X) cx . (2.183) 

^m. 



We can now interpret eq.(2.181 ) as follows. Our preference for a model will be 
dictated by how well the model fits the data; this is measured by [p(X|m, 9m)\^^^ 
Laiax- The volume of the region of uncertainty 69m also contributes: if more 
values of the parameters are consistent with the data, then there are more ways 
the model agrees with the data, and the model is favored. Finally, the larger the 
volume of possible parameter values Vm the more the model is penalized. Since 
a larger volume Vm means a more complex model the \/Vm factor penalizes 
complexity. The preference for simpler models is said to implement Occam's 
razor. This is a reference to the principle, stated by William of Occam, a 13th 
century Franciscan monk, that one should not seek a more complicated expla- 
nation when a simpler one will do. Such an interpretation is satisfying but 
ultimately it is quite unnecessary. Occam's principle does not need not be put 
in by hand: Bayes' theorem takes care of it automatically in eq.(2.181 1! 



2.11.4 Maximum Likelihood 

If one adopts the frequency interpretation of probabilities then most uses of 
Bayes' theorem are not allowed. The reason is simple: it makes sense to assign a 
probability distribution p{x\9) to the data X — {Xi\ because the x are random 
variables but it is absolutely meaningless to talk about probabilities for the 
parameters 9 because they have no frequency distributions, they are not random 
variables, they are merely unknown. This means that many problems in science 
lie beyond the reach of a frequentist probability theory. 

To overcome this difficulty a new subject was invented: statistics. Within 
the Bayesian approach the two subjects, statistics and probability theory, are 
unified into the single field of inductive inference. In the frequentist approach to 
statistics in order to infer an unknown quantity 9 on the basis of measurements 
of another quantity, the data x, one postulates the existence of some function, 
called the 'statistic', that relates the two, 9 = f{x). Since data are afflicted by 
experimental errors they are deemed to be legitimate random variables to which 
frequentist probability concepts can be applied. The problem is to estimate 
the unknown 9 when the sampling distribution p{x\9) is known. The solution 
proposed by Fisher was to select that value of 9 that maximizes the probability 
of the data that was actually obtained in the experiment. Since p{x\9) is a 
function of the variable x and 9 appears as a fixed parameter, Fisher introduced 
a function of 9, which he called the likelihood, where the observed data X appear 
as fixed parameters, 

L{9\X)=p{X\9). (2.184) 
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Thus, this method of parameter estimation is called the method of 'maximum 
likelihood'. The likelihood function L{9\X) is not a probability, it is not nor- 
malized in any way, and it makes no sense to use it compute an average or a 
variance, but the same intuition that leads one to propose maximization of the 
likelihood to estimate 9 also leads one to use the width of the likelihood function 
as to estimate an error bar. 

The Bayesian approach agrees with the method of maximum likelihood in 
the special case where of prior is uniform, 

p{e) = const p{0\X) oc p{0)p{X\0) oc p{X\0) . (2.185) 

This explains why the Bayesian discussion of this section has reproduced so 
many of the standard results of the 'orthodox' theory. But then there are ad- 
ditional advantages. Unlike the likelihood, the posterior is a true probability 
distribution that allows estimation not just of 9 but of any one of its moments. 
And, most important, there is no limitation to uniform priors. If there is ad- 
ditional prior information that is relevant to a problem the prior distribution 
provides a mechanism to take it into account. 



Chapter 3 



Entropy I: The Evolution of 
Carnot's Principle 

An important problem that occupied the minds of many scientists in the 18th 
century was either to devise a perpetual motion machine, or to prove its impos- 
sibility from the established principles of mechanics. Both attempts failed. Ever 
since the most rudimentary understanding of the laws of thermodynamics was 
achieved in the 19th century no competent scientist would waste time consid- 
ering perpetual motionj^ The other goal has also proved elusive; there exist no 
derivations the Second Law from purely mechanical principles. It took a long 
time, and for many the subject is still controversial, but the reason has gradually 
become clear: entropy is not a physical quantity, it is a tool for inference, a tool 
for reasoning in situations of incomplete information. It is quite impossible that 
such a non-mechanical quantity could emerge from a combination of mechanical 
notions. If anything it should be the other way around. 

Much of the material including the title for this chapter is inspired by a 
beautiful article by E. T. Jaynes [Jaynes 88]. I also borrowed from the historical 
papers [Klein 70, 73, Uffink 04]. 

3.1 Carnot: reversible engines 

Sadi Carnot was interested in improving the efficiency of steam engines, that 
is, of maximizing the amount of useful work that can be extracted from an 
engine per unit of burnt fuel. His work, published in 1824, was concerned with 
whether appropriate choices of a working substance other than steam and of the 

^The science of thermodynamics which led to statistical mechanics and eventually to infor- 
mation theory was initially motivated by the desire to improve steam engines. There seems to 
exist a curious historical parallel with the modern day development of quantum information 
theory, which is being driven by the desire to build quantum computers. The usefulness of 
thermodynamics far outgrew its original aim. It is conceivable that the same will happen to 
quantum information theory. 
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operating temperatures and pressures would improve the efficiency. 

Carnot was quite convinced that perpetual motion was impossible even 
though he had no proof. He could not have had a proof: thermodynamics 
had not been invented yet. His conviction derived from the long list of previous 
attempts that had ended in failure. His brilliant idea was to proceed anyway 
and to postulate what he knew was true but could not prove as the foundation 
from which he would draw all sorts of other conclusions about enginesj^ 

At the time Carnot did his work the nature of heat as a form of energy 
had not yet been understood. He adopted a model that was fashionable at the 
time - the caloric model - according to which heat is a substance that could be 
transferred but neither created nor destroyed. For Carnot an engine used heat 
to produce work in much the same way that falling water can turn a waterwheel 
and produce work: the caloric would "fall" from a higher temperature to a lower 
temperature thereby making the engine turn. What was being transformed into 
work was not the caloric itself but the energy acquired in the fall. 

According to the caloric model the amount of heat extracted from the high 
temperature source should be the same as the amount of heat discarded into 
the low temperature sink. Later measurements showed that this was not true, 
but Carnot was quite lucky. Although the model was seriously wrong, it did 
have a great virtue: it suggested that the generation of work in a heat engine 
should include not just the high temperature source from which heat is extracted 
(the boiler) but also a low temperature sink (the condenser) into which heat is 
discarded. Later, when heat was interpreted as a form of energy transfer it was 
understood that for continued operation it was necessary that excess heat be 
discarded into a low temperature sink so that the engine could complete each 
cycle by returning to same initial state. 

Carnot 's caloric- waterwheel model was fortunate in yet another respect - he 
was not just lucky, he was very lucky - a waterwheel engine can be operated in 
reverse and used as a pump. This led him to consider a reversible heat engine 
in which work would be used to draw heat from a cold source and 'pump it up' 
to deliver heat to the hot reservoir. The analysis of such reversible heat engines 
led Carnot to the important conclusion 

Carnot's Principle: "No heat engine E can be more efficient than a reversible 
one Eft operating between the same temperatures." 

The proof of Carnot's principle is quite straightforward but because he used 
the caloric model Carnot's proof was not strictly correct ~ the necessary revisions 
were supplied by Clausius in 1850. As a side remark, it is interesting that 
Carnot's notebooks, which were made public by his family about 1870, long 
after his death, indicate that soon after 1824 Carnot came to reject the caloric 

^In his attempt to understand the undetectabihty of the ether Einstein faced a similar 
problem: he knew that it was hopeless to seek an understanding of the constancy of the speed 
of light on the basis of the primitive physics of the atomic structure of solid rods that was 
available at the time. Inspired by Carnot he deliberately followed the same strategy - to give 
up and declare victory - and postulated the constancy of the speed of light as the unproven 
but known truth which would serve as the foundation from which other conclusions could be 
derived. 
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model and that he achieved the modern understanding of heat as a form of 
energy transfer. This work - which had preceded Joule's experiments by about 
fifteen years - was not published and therefore had no influence on the history 
of thermodynamics [Wilson 81]. 

The following is Clausius' proof. In a standard cycle (Figure 3.1a) a heat 
engine E extracts heat qi from a reservoir at high temperature ti and partially 
converts it to useful work w. The difference qi — w — q2 is wasted heat that 
is dumped into a reservoir at a lower temperature i2- The Carnot-Clausius 
argument is that if an engine Es exists that is more efficient than a reversible 
engine Er, then it is possible to build perpetual motion machines. Since the 
latter do not exist Carnot's principle follows: heat engines that are more efficient 
than reversible ones do not exist. 
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Figure 3.1: (a) A regular engine E operating between heat reservoirs at tem- 
peratures ti and ^2 generates work w = q\ — q2- (b) A (hypothetical) super- 
efficient engine Eg linked to a reversed engine E^. would be a perpetual mo- 
tion engine extracting heat from the cold reservoir and converting it to work 

Ws ~Wr^ q2B. - q2S- 



Consider two engines, one is super-efficient and the other is reversible, Es 
and Eji, operating between the same hot and cold reservoirs. The engine Eg 
draws heat qi from the hot source, it generates work ws, and delivers the dif- 
ference as heat q2s = qi — ws to the cold sink (figure 3.1b). It is arranged 
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that in its normal (forward) operation the reversible engine En draws the same 
heat qi from the hot source, it generates work wr, and discards the difference 
Q2R = qi — Wfi to the cold sink. Since Es is supposed to be more efficient than 
E}i we have ws > wr, it would be possible to use a part wr of the work pro- 
duced by Eg to run Er in reverse. The result would be to extract heat q2R from 
the cold source and pump the total heat q2R + wr — qi back up into the hot 
source. The remaining work ws — wr produced by Eg would then be available 
for any other purposes. At the end of such composite cycle the hot reservoir is 
left unchanged and the net result would be to extract heat q2R — q2 > from 
the cold reservoir and convert it to work ws — wr without any need for fuel. 
The conclusion is that the existence of a super-efficient heat engine would allow 
the construction of a perpetual motion engine. 

The blank statement perpetual motion is not possible is a true principle but 
it does not tell the whole story. It blurs the important distinction between 
perpetual motion engines that operate by violating energy conservation, which 
are called machines of the first kind, and perpetual motion engines that do not 
violate energy conservation, which are thus called machines of the second kind. 
Carnot's conclusion deserves to be singled out as a new principle because it is 
specific to the second kind of machine. 

Other important conclusions obtained by Carnot are that all reversible en- 
gines operating between the same temperatures are equally efficient; their effi- 
ciency is a function of the temperatures only, 

e''J^'^ = e{t^,t2), (3.1) 

and is therefore independent of any and all other details of how the engine is 
constructed and operated; that efficiency increases with the temperature differ- 



ence [see eq.(3.3l below]. Furthermore, the most efficient heat engine cycle, now 
called the Carnot cycle, is one in which all heat is absorbed at the high ti and 
all heat is discharged at the low t2- Thus, the Carnot cycle is defined by two 
isotherms and two adiabats. 

The next important step, the determination of the universal function 6(^1,^2), 
was accomplished by Kelvin. 



3.2 Kelvin: temperature 

After Joule's experiments in the 1840's on the conversion of work into heat the 
caloric model had to be abandoned. Heat was finally recognized as a form of 
energy and the additional relation w — qi — q2 was the ingredient that, in the 
hands of Kelvin and Clausius, allowed Carnot's principle to be developed into 
the next stage. 

Suppose two reversible engines Ea and Ei, are linked in series to form a single 
more complex reversible engine Ec- The first operates between temperatures ti 
and t2, and the second between t2 and ^3. Ea draws heat qi and discharges q2, 
while Eb uses 52 as input and discharges q^ . The efficiencies of the three engines 
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are ^ ^ 

Ca = e(ii,i2) = — , eb = e{t2,h)^ — , (3.2) 
9i 92 



and 



They are related by 



or 



ec = e{ti,t3)^ . (3.3) 



Wbq2 , , \ a A\ 
= Ca + eb 1 , (3.4) 

92 91 V 91 



Cc = ea + eb - Cafib , (3.5) 
which is a functional equation for e = e(fi,t2)- To find the solution change 



variables to x = log (1 — e), which transforms eq.(3.5) into 

Xc{h,h) ^ Xa{tl,t2) + Xb{t2,h) , (3.6) 

and then differentiate with respect to ^2 to get 

9 , . 9 , , ,„ 

-^Xa{tl,t2) = -^Xh{t2,h) . (3.7) 
ai2 Ot2 

The left hand side is independent of while the second is independent of ti, 
therefore dxa/dt2 must be some function g of t2 only, 

-^^Xa{hM)=9{t2) ■ (3.8) 

Integrating gives x{ti,t2) = F{ti) + G{t2) where the two functions F and G 
are at this point unknown. The boundary condition e{t,t) — or equivalently 
x(t,t) = implies that we deal with merely one unknown function: G(t) — 
-F{t). Therefore 

x{h,t2) ^ F{h) - F{t2) or e(ti,i2) = l-^ , (3.9) 



where / = e~^. From eq.(3.3l we sec that the efficiency e{ti,t2) increases as 
the difference in temperature increases, so that / (t) must be a monotonically 
increasing function. 

Kelvin recognized that there is nothing fundamental about the original tem- 
perature scale t. It depends on the particular materials employed to construct 



the thermometer. Kelvin realized that the freedom in eq.(3.9l in the choice of 
the function / corresponds to the freedom of changing temperature scales by 
using different thermometric materials. The only feature common to all ther- 
mometers that claim to rank systems according to their 'degree of hotness' is 
that they must agree that if A is hotter than B, and B is hotter than C, then 
A is hotter than C. One can therefore regraduate any old inconvenient t scale 
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by a monotonic function to obtain a new scale T chosen purely because it leads 



to a more elegant formulation of the theory. From eq.(3.9l the optimal choice 



is quite obvious, and thus Kelvin introduced the absolute scale of temperature, 

T^Cfit) , (3.10) 

where the arbitrary scale factor C reflects the still remaining freedom to choose 
the units. In the absolute scale the efficiency for the ideal reversible heat engine 
is very simple, 

e{h,h) = l-^. (3.11) 

Carnot's principle that any heat engine E' must be less efficient than the 
reversible one, e' < e, is rewritten as 

f W 02 T2 

e = — = 1 <e=l-— , (3.12) 

qi qi Ti 

or, 

^ - ^ < . (3.13) 

It is convenient to redefine heat so that inputs are positive, Qi = qi, and outputs 
are negative, Q2 = —92- Then, 

^ + ^<0, (3.14) 
-1-1 J2 

where the equality holds when and only when the engine is reversible. 

The generalization to an engine or any system that undergoes a cyclic process 
in which heat is exchanged with more than two reservoirs is straightforward. If 
heat Qi is absorbed from the reservoir at temperature Ti we obtain the Kelvin 
form (1854) of Carnot's principle, 

E^<0. (3.15) 

which, in the hands of Clausius, led to the next non-trivial step, the introduction 
of the concept of entropy. 



3.3 Clausius: entropy 

By about 1850 both Kelvin and Clausius had realized that two laws were nec- 
essary as a foundation for thermodynamics. The somewhat awkward expres- 
sions for the second law that they had adopted at the time were reminiscent of 
Carnot's; they stated the impossibility of heat engines whose sole effect would 
be to transform heat from a single source into work, or of refrigerators that could 
pump heat from a cold to a hot reservoir without the input of external work. It 
took Clausius until 1865 - this is some fifteen years later, which indicates that 
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the breakthrough was not at all trivial - before he came up with a new compact 
statement of the second law that allowed substantial further progress. [Cropper 
86] 



Clausius rewrote Kelvin's eq.(3.15l for a cycle where the system absorbs in- 



finitesimal (positive or negative) amounts of heat dQ from a continuous sequence 
of reservoirs, 

/^<0, (3.16) 

where T is the temperature of each reservoir. For a reversible process, which 
is achieved when the system is slowly taken through a sequence of equilibrium 
states and T is the temperature of the system as well as the reservoirs, the 
equality sign implies that the integral from any state A to any other state B is 
independent of the path taken. 



dQ 







/ 

Ri{A,B 



dQ 
T 
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R2(A,B) 



dQ 
T 



(3.17) 



where Ri{A, B) and i?2(^, B) denote any two reversible paths linking the same 
initial state A and final state B. Clausius saw that this implied the existence of 
a function of the thermodynamic state, which he called the entropy, and defined 
up to an additive constant by 



Sb 



Sa+ J 

R(A.B, 



dQ 
T 



(3.18) 



At this stage in the development this entropy is 'thermodynamic entropy', and 
is defined only for equilibrium states. 



Eq.(3.18) seems like a mere reformulation of eqs.( 3.151 and (3.16) but it 
represents a major advance because it allowed thermodynamics to reach beyond 
the study of cyclic processes. Consider a possibly irreversible process in which 
a system is taken from an initial state A to a final state B, and suppose the 
system is returned to the initial state along some other reversible path. Then, 



the more general eq.(3.16l gives 



A,irrcv ^ R(A,B) ^ 



(3.19) 



From eq.(3.18l the second integral is Sa — Sb. In the first integral ~dQ is the 



amount is the amount of heat absorbed by the reservoirs at temperature T and 
therefore it represents minus the change in the entropy of the reservoirs, which 
in this case represent the rest of the universe. 



{ST - S'b) + {Sa - 5b) < or S^ + Sb> ST + Sa • (3.20) 



Thus the second law can be stated in terms of the total entropy S 
as 



ototal \, ctotal 
'-'final — '-'initial J 



total g\cs g 

(3.21) 
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and Clausius could then summarize the laws of thermodynamics as " The energy 
of the universe is constant. The entropy of the universe tends to a maximum." 
All restrictions to cyclic processes have disappeared. 

Clausius was also responsible for initiating another independent line of re- 
search in this subject. His paper "On the kind of motion we call heat" (1857) 
was the first (failed!) attempt to deduce the second law from purely mechanical 
principles applied to molecules. His results referred to averages taken over all 
molecules, for example the kinetic energy per molecule, and involved theorems 
in mechanics such as the virial theorem. For him the increase of entropy was 
meant to be an absolute law and not just a matter of overwhelming probability. 

3.4 Maxwell: probability 

We owe to Maxwell the introduction of probabilistic notions into fundamental 
physics (1860). (Perhaps he was inspired by his earher study of the rings of 
Saturn which required reasoning about particles undergoing very complex tra- 
jectories.) He realized the impossibility of keeping track of the exact motion of 
all the molecules in a gas and pursued a less detailed description in terms of the 
distribution of velocities. Maxwell interpreted his distribution function as the 
fraction of molecules with velocities in a certain range, and also as the "proba- 
bility" P{v)cPv that a molecule has a velocity v va & certain range d^v. It would 
take a long time to achieve a clearer understanding of the meaning of the term 
'probability'. In any case, Maxwell concluded that "velocities are distributed 
among the particles according to the same law as the errors are distributed in 
the theory of the 'method of least squares'," and on the basis of this distribution 
he obtained a number of significant results on the transport properties of gases. 

Over the years he proposed several derivations of his velocity distribution 
function. The earlier one (1860) is very elegant. It involves two assumptions: 
the first is a symmetry requirement, the distribution should only depend on the 
actual magnitude j"?! = w of the velocity and not on its direction, 

P{v)d^v = P (^^vl + vl + vi) d^v . (3.22) 

The second assumption is that velocities along orthogonal directions should be 
independent 

P{v)d^v = p{v,)p{vy)p{v,)d^v . (3.23) 

Therefore 

P [sjvl + vl + vi) = p{v,)p{vy)p{v,) . (3.24) 
Setting Vy = V;^ = we get 

P{v.)=pMp{0)piO) , (3.25) 
so that we obtain a functional equation for p, 



P vl + vl + vfj p{0)p{0) = Piv^)p{vy)p{v;,) , 



(3.26) 
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or 



log 



p(0) 



= log 



p(0) 



log 



p(0) 



log 



P{Vz) 

p(0) 



or, introducing the functions G, 

G (^^vl + vl + vi) = GK) + G{vy) + G{v,). 



(3.27) 



(3.28) 



The solution is straightforward. Differentiate with respect to Vx and to Vy to 
get 



-Vx = G [Vx) and - 



vl + vl + vl 



or 



G'{Vx) _ G'{Vy) 



= -2a , 



where —2a is a constant. Integrating gives 



so that 



log 



P{v) 



pM 



p{0) 



Givx) 



■ const 



exp [—a (i 



G'ivx) , 
(3.29) 
(3.30) 

(3.31) 
(3.32) 



the same distribution as "errors in the method of least squares" . 

Maxwell's distribution applies whether the molecule is part of a gas, a liquid, 
or a solid and, with the benefit of hindsight, the reason is quite easy to see. 
The probability that a molecule have velocity v and position x is given by 
the Boltzmann distribution oc cxp—H/kT. For a large variety of situations 
the Hamiltonian for one molecule is of the form H = + V{x) where 

the potential V{x) includes the interactions, whether they be weak or strong, 
with all the other molecules. If the potential V{x) is independent of v, then 
the distribution for v and x factorizes. Velocity and position are statistically 
independent, and the velocity distribution is Maxwell's. 

Maxwell was the first to realize that the second law is not an absolute law 
(this was expressed in his popular textbook 'Theory of Heat' in 1871), that it 
'has only statistical certainty' and indeed, that in fiuctuation phenomena 'the 
second law is continually being violated'. Such phenomena are not rare: just 
look out the window and you can see the sky is blue - a consequence of the 
scattering of light by density fluctuations in the atmosphere. 

Maxwell introduced the notion of probability, but what did he actually mean 
by the word 'probability' ? He used his distribution function as a velocity dis- 
tribution, the number of molecules with velocities in a certain range, which 
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betrays a frcqucntist interpretation. These probabilities are ultimately mechan- 
ical properties of the gas. But he also used his distribution to represent the 
lack of information we have about the precise microstate of the gas. This lat- 
ter interpretation is particularly evident in a letter he wrote in 1867 where he 
argues that the second law could be violated by "a finite being who knows the 
paths and velocities of all molecules by simple inspection but can do no work 
except open or close a hole." Such a "demon" could allow fast molecules to pass 
through a hole from a vessel containing hot gas into a vessel containing cold 
gas, and could allow slow molecules pass in the opposite direction. The net 
effect being the transfer of heat from a low to a high temperature, a violation 
of the second law. All that was required was that the demon "know" the right 
information. [Klein 70] 

3.5 Gibbs: beyond heat 

Gibbs generalized the second law in two directions: to open systems and to 
inhomogeneous systems. With the introduction of the concept of the chemical 
potential, a quantity that regulates the transfer of particles in much the same 
way that temperatTire regulates tlw^ transfer of heat, he could apply the meth- 
ods of thermodynamics to phase transitions, mixtures and solutions, chemical 
reactions, and much else. His paper "On the Equilibrium of Heterogeneous Sys- 
tems" [Gibbs 1875-78] is formulated as the purest form of thermodynamics - 
a phenomenological theory of extremely wide applicability because its founda- 
tions do not rest on particular models about the structure and dynamics of the 
microscopic constituents. 

And yet, Gibbs was keenly aware of the significance of the underlying molec- 
ular constitution - he was familiar with Maxwell's writings and in particular 
with his "Theory of Heat" (indeed, he found mistakes in it). His discussion of 
the process of mixing gases led him to analyze the "paradox" that bears his 
name. The entropy of two different gases increases when the gases are mixed; 
but does the entropy also increase when two gases of the same molecular species 
are mixed? Is this an irreversible process? 

For Gibbs there never was a 'paradox', much less one that would require 
some esoteric new (quantum) physics for its resolution. For him it was quite 
clear that thermodynamics was not concerned with microscopic details, but 
rather with the changes from one macrostate to another. He explained that the 
mixing of two gases of the same molecular species cannot be reversed because 
the mixing does not lead to a different "thermodynamic" state; 

". . .we do not mean a state in which each particle shall occupy more or less 
exactly the same position as at some previous epoch, but only a state which 
shall be indistinguishable from the previous one in its sensible properties. 
It is to states of systems thus incompletely defined that the problems of 
thermodynamics relate." [Gibbs 1875-78] 
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Gibbs' resolution of the paradox hinges on recognizing, as had Maxwell be- 
fore him, that the explanation of the second law cannot rest on purely me- 
chanical arguments, that probabilistic concepts are required. This led him to 
conclude: "In other words, the impossibility of an uncompensated decrease 
of entropy seems to be reduced to improbability," a sentence that Boltzmann 
adopted as the motto for the second volume of his "Lectures on the Theory of 
Gases." (For a modern discussion of the Gibbs' paradox see section 4.12.) 

Remarkably neither Maxwell nor Gibbs established a connection between 
probability and entropy. Gibbs was very successful at showing what one can 
accomplish by maximizing entropy but he did not address the issue of what 
entropy is or what it means. The crucial steps in this direction were taken by 
Boltzmann. 

But Gibbs' contributions did not end here. The ensemble theory introduced 
in his "Principles of Statistical Mechanics" in 1902 (it was Gibbs who coined the 
term 'statistical mechanics') represent a practical and conceptual step beyond 
Boltzmann's understanding of entropy. 

3.6 Boltzmann: entropy and probability 

It was Boltzmann who found the connection between entropy and probability, 
but his path was long and tortuous [Klein 73, Ufhnk 04]. Over the years he 
adopted several different interpretations of probability and, to add to the con- 
fusion, he was not always explicit about which one he was using, sometimes 
mixing them within the same paper, and even within the same equation. At 
first, he defined the probability of a molecule having a velocity v within a small 
cell dPv as being proportional to the amount of time that the particle spent 
within that particular cell, but he also defined that same probability as the 
fraction of particles within the cell. 

By 1868 he had managed to generalize the Maxwell distribution for point 
particles and the theorem of equipartition of energy to complex molecules in the 
presence of an external field. The basic argument, which led him to the Boltz- 
mann distribution, was that in equilibrium the distribution should be stationary, 
that it should not change as a result of collisions among particles. 

The collision argument only gave the distribution for individual molecules; 
it was also in 1868 that he first applied probability to the system as a whole 
rather than to the individual molecules. He identified the probability of the 
system being in some region of the A'^-particle phase space (rather than the 
space of molecular velocities) with the relative time the system would spend 
in that region - the so-called "time" ensemble. Alternatively, probability was 
also defined at a given instant in time as being proportional to the volume of 
the region. At first he did not think it was necessary to comment on whether 
the two definitions are equivalent or not, but eventually he realized that their 
'probable' equivalence should be explicitly expressed as the hypothesis, which 
later came to be known as the 'ergodic; hypothesis', that over a long time the 
trajectory of the system would cover the whole region of phase space consistent 
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with the given value of the energy. At the time all these probabilities were still 
conceived as mechanical properties of the gas. 

In 1871 Boltzmann achieved a significant success in establishing a connection 
between thermodynamic entropy and microscopic concepts such as the proba- 
bility distribution in phase space. In modern notation his argument was as 
follows. The energy of N interacting particles is given by 

N „2 

H = J: ^ + U{xi,...,xn) ■ (3.33) 

The first non-trivial decision was to specify what quantity defined in purely 
microscopic terms corresponds to the macroscopic internal energy. He opted for 
the "average" 

E=(H) = J dzN Pn H , (3.34) 

where dzN — d^^ xd^^ p is the volume element in the A^-particle phase space, 
and P^v is the A^-particle distribution function, 

Pn = ''''P^;^^^ where Z = ^dzj, e'^" , (3.35) 
and /3 — l/kT, so that, 

E = ^iVfcT-l- ([/) . (3.36) 

The connection to the thermodynamic entropy requires a clear idea of the 
nature of heat and how it differs from work. One needs to express heat in purely 
microscopic terms, and this is quite subtle because at the molecular level there is 
no distinction between thermal motions and just plain motions. The distribution 
function is the crucial ingredient. In any infinitesimal transformation the change 
in the internal energy separates into two contributions, 

5E = ^ dzN H5Pn + J dzN PnSH . (3.37) 

The second integral, which can be written as (SH) ~ {^U), arises purely from 
changes in the potential function U, which depends among other things on 
the volume of the vessel containing the gas. Now, a change in the potential 
is precisely what one means by mechanical work 6W, therefore, since 6E — 
SQ + SW, the first integral must represent the transferred heat SQ, 

5Q = 5E- (SU) . (3.38) 



On the other hand, substituting SE from eq. (3.361, one gets 



SQ='^NkST + 5{U)- {SU) . (3.39) 



This is not a complete differential, but dividing by the temperature yields 

(3.40) 



SQ 
T 



^NklogT+ ^ + klog{Jd^^xe-''^'^) +const 
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which suggests that the expression in brackets should be identified with the 
thermodynamic entropy S. Further rewriting leads to 

E 

S = — + klogZ + const, (3.41) 

which is recognized as the correct modern expression. 

Boltzmann's path towards understanding the second law was guided by one 
notion from which he never wavered: matter is an aggregate of molecules. Apart 
from this the story of his progress is the story of the increasingly more impor- 
tant role played by probabilistic notions, and ultimately, it is the story of the 
evolution of his understanding of the notion of probability itself. By 1877 Boltz- 
mann achieves his final goal and explains entropy purely in terms of probability 
- mechanical notions were by now reduced to the bare minimum consistent with 
the subject matter: we are, after all, talking about collections of molecules and 
their energy is conserved. His final achievement hinges on the introduction of 
yet another way of thinking about probabilities. 

He considered an idealized system consisting of N particles whose single- 
particle phase space is divided into rn cells each with energy £„, n — 1, ...,m. 
The number of particles in the nth cell is denoted Wn, and the distribution 
'function' is given by the set of numbers Wi, . . . ,Wm- In Boltzmann's previous 
work the determination of the distribution function had been based on figur- 
ing out its time evolution from the mechanics of collisions. Here he used a 
purely combinatorial argument. A completely specified state, which he called a 
complexion, and we call a microstate, is defined by specifying the cell of each 
individual molecule. A macrostate is less completely specified by the distribu- 
tion function, Wi,. . . , Wm- The number of microstates compatible with a given 
macrostate, which he called the 'permutability', and we call the 'multiplicity' is 

N\ 

W = — . (3.42) 

Boltzmann's assumption was that the probability of the macrostate was propor- 
tional to its multiphcity, to the number of ways in which it could be achieved, 
which assumes each microstate is as likely as any other - the 'equal a priori 
probability postulate'. 

The most probable macrostate is that which maximizes W subject to the 
constraints of a fixed total number of particles N and a fixed total energy E, 

m m 

J2wn = N and J2wn£n = E. (3.43) 

71—1 n=l 

When the numbers Wn are large enough that one can use StirHng's approxima- 
tion for the factorials, we have 

m 

logW = NlogN - N - {wn log Wn-Wn) (3.44) 

n=l 

m 

= - w„ log Wn + const , (3.45) 

n=l 
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or perhaps better 



logM/ = -ivg^log^ (3.46) 

n=l 



SO that 

m 

logW' = -iVS/„log/„ (3.47) 

where /„ — Wn/N is the fraction of molecules in the nth cell with energy e„, or, 
alternatively the probability that a molecule is in its nth state. The distribution 



that maximizes logVF subject to the constraints (3.431 is such that 



/„ = ^ (X e-^^" , (3.48) 

where /3 is a Lagrange multiplier determined by the total energy. When applied 
to a gas, the possible states of a molecule are cells in phase space. Therefore 

logW^~N J dzi f{x,p) log fix,p) , (3.49) 

where dzi = (PxcPp and the most probable distribution is the equilibrium dis- 
tribution found earlier by Maxwell and generalized by Boltzmann. 

In this approach probabilities are central. The role of dynamics is minimized 
but it is not eliminated. The Hamiltonian enters the discussion in two places. 
One is quite explicit: there is a conserved energy the value of which is imposed as 
a constraint. The second is much more subtle; we saw above that the probability 
of a macrostate could be taken proportional to the multiplicity W provided 
microstates are assigned equal probabilities, or equivalently, equal volumes in 
phase space are assigned equal a priori weights. As always equal probabilities 
must be justified in terms of some form of underlying symmetry. In this case, the 
symmetry follows from Liouville's theorem - under a Hamiltonian time evolution 
a region in phase space will move around and its shape will be distorted but 
its volume will be conserved; Hamiltonian time evolution preserves volumes in 
phase space. The nearly universal applicability of the 'equal a priori postulate' 
can be traced to the fact that what is needed is a Hamiltonian; any Hamiltonian 
would do. 

It is very remarkable that although Boltzmann calculated the maximized 
value log W for an ideal gas and knew that it agreed with the thermodynamical 
entropy except for a scale factor, he never wrote the famous equation that bears 
his name 

S=klogW. (3.50) 

This equation, as well as Boltzmann's constant k, were both first written by 
Planck. 



There is, however, a problem with eq.(3.49l: it involves the distribution 



function f{x,p) in the one-particle phase space and therefore it cannot take 



correlations into account. Indeed, eq.(3.49 1 gives the correct form of the entropy 
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only for ideal gases of non-interacting particles. The expression that applies to 
systems of interacting particles i^ 



logW ^ ~ / dzM Jn log /at , 



(3.51) 



where /tv = fN{xi,pi,...,XN,PN) is the probability distribution in the iV- 
particle phase space. This equation is usually associated with the name of Gibbs 
who, in his "Principles of Statistical Mechanics" (1902), developed Boltzmann's 
combinatorial arguments into a very powerful theory of ensembles. The con- 



ceptual gap between eq.(3.49) and (3.51 1 is enormous; it goes well beyond the 



issue of intermolecular interactions. The probability in Eq.(3.49) is the single- 



particle distribution, it can be interpreted as a "mechanical" property, namely. 



the relative number of molecules in each cell. The entropy Eq.(3.49l is a me- 



chanical property of the individual system. In contrast, eq.(3.51l involves the 



A^-particle distribution which is not a property of any single individual system 
but a property of an ensemble of replicas of the system. Gibbs was not very 
explicit about his interpretation of probability. He wrote 

"The states of the bodies which we handle are certainly not known to us 
exactly. What we know about a body can generally be described most 
accurately and most simply by saying that it is one taken at random from 
a great number (ensemble) of bodies which are completely described." [my 
italics, Gibbs 1902, p. 163] 

It is clear that for Gibbs probabilities represent a state of knowledge, that the 
ensemble is a purely imaginary construction, just a tool for handling incomplete 
information. On the other hand, it is also clear that Gibbs still thinks of prob- 
abilities in terms of frequencies, and since the actual replicas of the system do 
not exist, he is forced to imagine them. 

This brings our story of entropy up to about 1900. In the next chapter we 
start a more deliberate and systematic study of the connection between entropy 
and information. 



3.7 Some remarks 

I end with a disclaimer: this chapter has historical overtones but it is not history. 
Lines of research such as the Boltzmann equation and the ergodic hypothesis 
that were historically very important have been omitted because they represent 
paths that diverge from the central theme of this work, namely how laws of 
physics can be derived from rules for handling information and uncertainty. Our 
goal has been and will be to discuss thermodynamics and statistical mechanics 
as the first historical example of such an information physics. At first I tried to 
write a 'history as it should have happened'. I wanted to trace the development 



•^For the moment we disregard the question of the distinguishabiUty of the molecules. The 
so-called Gibbs paradox and the extra factor of 1/A'^! will be discussed in detail in chapter 4. 
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of the concept of entropy from its origins with Carnot in a manner that reflects 
the logical rather than the actual evolution. But I found that this approach 
would not do; it trivializes the enormous achievements of the 19th century 
thinkers and it misrepresents the actual nature of research. Scientific research 
is not a tidy business. 

I mentioned that this chapter was inspired by a beautiful article by E. T. 
Jaynes with the same title [Jaynes 88]. I think Jaynes' article has great peda- 
gogical value but I disagree with him on how well Gibbs understood the logical 
status of thermodynamics and statistical mechanics as examples of inferential 
and probabilistic thinking. My own assessment runs in quite the opposite di- 
rection; the reason why the conceptual foundations of thermodynamics and 
statistical mechanics have been so controversial throughout the 20th century is 
precisely because neither Gibbs nor Boltzmann were particularly clear on the 
interpretation of probability. I think that we could hardly expect them to have 
done much better; they did not benefit from the writings of Keynes (1921), 
Ramsey (1931), de Finetti (1937), Jeffreys (1939), Cox (1946), Shannon (1948), 
Polya (1954) and, of course, Jaynes himself (1957). Indeed, whatever clarity 
Jaynes attributes to Gibbs, is not Gibbs'; it is the hard-won clarity that Jaynes 
attained through his own efforts and after absorbing much of the best the 20th 
century had to offer. 



Chapter 4 



Entropy II: Measuring 
Information 



What is information? Our central goal is to gain insight into the nature of 

information, how one manipulates it, and the implications of such insights for 
physics. In chapter 2 we provided a first partial answer. We might not yet know 
precisely what information is, but we know it when we see it. For example, it 
is clear that experimental data contains information, that it is processed using 
Bayes' rule, and that this is very relevant to the empirical aspect of science, 
namely, to data analysis. Bayes' rule is the machinery that processes the in- 
formation contained in data to update from a prior to a posterior probability 
distribution. This suggests the following generalization; "information" is what- 
ever induces one to update from one state of belief to another. This is a notion 
worth exploring and to which we will return later. 

In this chapter we pursue another point of view that has turned out to be 
extremely fruitful. We saw that the natural way to deal with uncertainty, that 
is, with lack of information, is to introduce the notion of degrees of belief, and 
that these measures of plausibihty should be manipulated and calculated using 
the ordinary rules of the calculus of probabilities. But with this achievement 
we do not yet reach our final goal. The rules of probability theory allow us to 
assign probabilities to some "complex" propositions on the basis of the proba- 
bilities that have been previously assigned to other, perhaps more "elementary" 
propositions. 

In this chapter we introduce a new inference tool designed specifically for 
assigning those elementary probabilities. The new tool is Shannon's measure of 
an "amount of information" and the associated method of reasoning is Jaynes' 
Method of Maximum Entropy, or MaxEnt. [Shannon 48, Jaynes 57b, 83, 03] 
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4.1 Shannon's information measure 

We appeal once more to the idea that if a general theory exists it must apply to 
special cases. Consider a set of mutually exclusive and exhaustive alternatives 
i, for example, the possible values of a variable, or the possible states of a 
system. The state of the system is unknown. On the basis of the incomplete 
information / we have we can at best assign probabilities p{i\I) = Pi- In order 
to select just one among the possible states more information is required. The 
question we address here is how much more? Note that we are not asking the 
more difficult question of which particular piece of information is missing, but 
merely the quantity that is missing. It seems reasonable that the amount of 
information that is missing in a sharply peaked distribution is smaller than the 
amount missing in a broad distribution, but how much smaller? Is it possible to 
quantify the notion of amount of information? Can one find a unique quantity 
S that is a function of the pi's, that tends to be large for broad distributions 
and small for narrow ones? 

Consider a discrete set of n mutually exclusive and exhaustive discrete al- 
ternatives i, each with probability Pi. According to Shannon, any measure S 
of the amount of information that is missing when all we know is a probability 
distribution must satisfy three axioms. It is quite remarkable that these con- 
ditions are sufficiently constraining to determine the quantity S uniquely. The 
first two axioms are deceptively simple. 

Axiom 1. 5 is a real continuous function of the probabilities pi, S[p] = 

S {Pl, ■ ■ -Pn)- 

Remark: It is explicitly assumed that S\p] depends only on the Pi and on 
nothing else. What we seek here is an absolute measure of the amount of 
missing information in p. If the objec;tive were to update from a prior q to a 
posterior distribution p - a problem that will be later tackled in chapter 6 - 
then we would require a functional S\p,q] depending on both q and p. Such 
S\p, q\ would at best be a relative measure: the information in p relative to the 
reference distribution q. 

Axiom 2. If all the piS are equal, pi = 1/n. Then S = S {1/n, . . . ,1/n) = 
F (n) , where F (n) is an increasing function of n. 

Remark: This means that it takes less information to pinpoint one alternative 
among a few than among many and also that knowing the number n of available 
states is already a valuable piece of information. Notic;e that the uniform distri- 
bution Pi = 1/n is singled out to play a very special role. Indeed, although no 
reference distribution has been explicitly mentioned, the uniform distribution 
will, in effect, provide the standard of complete ignorance. 

The third axiom is a consistency requirement and is somewhat less intuitive. 
The entropy S[p] measures the amount of additional information beyond the in- 
complete information / already codified in the pi that will be needed to pinpoint 
the actual state of the system. Imagine that this missing information were to 
be obtained not all at once, but in installments. The consistency requirement 
is that the particular manner in which we obtain this information should not 
matter. This idea can be expressed as follows. 
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Imagine the n states are divided into N groups labeled hy g = 1, . . . , N. The 
probability that the system is found in group g is 

Pg = T.p^- (4-1) 

Let p^g denote the conditional probability that the system is in the state i G g 
given it is in group g, 

Pi\9 = ^ for iGg. (4.2) 

Suppose we were to obtain the desired information in two steps, the first of 
which would allow us to single out one of the groups g while the second would 
allow us to decide on the actual i within the selected group g. The amount 
of information required in the first step is Sq = S[P] where P = {Pg} with 
g = 1 . . . N. Now suppose we did get this information, and as a result we found, 
for example, that the system was in group gi. Then for the second step, to 
single out the state i within the group gi, the amount of additional information 
needed would be Sg^ = S[p.\g^]. Similarly, information amounts Sg^jSg^, ... or 
Sgj^ would be required had the selected groups turned out to be g2, .93, . . . or gjv- 
But at the beginning of this process we do not yet know which of the g's is the 
correct one. The expected amount of missing information to take us from the 
g's to the actual i's is PgSg- The point is that it should not matter whether 
we get the total missing information in one step, which completely determines i, 
or in two steps, the first of which has low resolution and only determines one of 
the groups, say g, while the second step provides the fine tuning that determines 
i within the given g. This gives us our third axiom; 

Axiom 3. For all possible groupings g = 1 . . .N of the states i = 1 . . . n we 
must have 

S = Sg + j:P,Sg . (4.3) 
g 

This is called the "grouping" property. 

Remark: Given axiom 3 it might seem more appropriate to interpret 5 as a 
measure of the expected rather than the actual amount of missing information, 
but if 5* is the expected value of something, it is not clear, at this point, what 
that something would be. We will return to this below. 

The solution to Shannon's constraints is obtained in two steps. First assume 
that all states i are equally likely, pi = 1/n. Also assume that the N groups g 
all have the same number of states, m = n/N, so that Pg = 1/N and p^g = 
Pi/Pg = 1/m. Then by axiom 2, 



and 



S\pi] = S{l/n,... 
Sg[P,] = S{1/N,.. 

Sg\Pi\g] = S{l/m,.. 



.,l/n)=F{n), 
..,1/N) = F{N), 

..,1/to) = F{m). 



(4.4) 
(4.5) 

(4.6) 
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Then, axiom 3 gives 

F{mN) = F{N) + F{m) . (4.7) 

This should be true for all integers A'^ and m. It is easy to see that one solution 
of this equation is 

F{m) = k log m , (4.8) 



where k is any positive constant, but it is easy to see that eq.(4.7) has infinitely 



many other solutions. Indeed, since any integer m can be uniquely decomposed 
as a product of prime numbers, m — Jlr?" ' i where ai are integers and qr are 
prime numbers, using eq.(4.7l we have 



F{m)=j:^arF{qr) 



(4.9) 



which means that eq.( |4.7| can be satisfied by arbitrarily specifying F(qr) on the 
primes and then defining F{m) for any other integer through eq.(4.9l. 

A unique solution is obtained when we impose the additional requirement 
that F(m) be monotonic increasing in m (axiom 2). The following argument is 
found in [Jayncs 03] . Consider any two integers s and t both larger than 1 . The 
ratio of their logarithms can be approximated arbitrarily closely by a rational 
number, i.e., we can find integers a and (3 (with f3 arbitrarily large) such that 



a log s a + 1 „ a „ , 1 

[3 - logi /3 

But F is monotonic increasing, therefore 

F{e) < F(/) < i^(t"+i) , 



and using eq.(4.7l 



aF{t) < l3F{s) < {a + l)F{t) 



a F{s) a + 



(4.10) 



(4.11) 



(4.12) 



Which means that the ratio F(r)/ F(s) can be approximated by the same ratio- 



nal number a/f3. Indeed, comparing eqs.(4.10l and (4.12) we get 



m 



logs 
lost 



< 



1 



or, 



F{s) F{t) 



< 



m 

Plogs 



(4.13) 



(4.14) 



log s log t 

We can make the right hand side arbitrarily small by choosing P sufficiently 
large, therefore i^(s)/logs must be a constant, which proves (4.8) is the unique 
solution. 

In the second step of our derivation we will still assume that all is are equally 
likely, so that pi = l/n and S[p\ — F (n). But now we assume the groups g have 
different sizes, nig, with Pg 



nig/n and p^g = \/mg. Then axiom 3 becomes 



F{n)^SG[P] + Y.PgF{mg), 
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Therefore, 

Sg[P] = F (n) - ^ P,F (m,) =^PAF (n) - F (m,)] . 
a 9 

Substituting our previous expression for F we get 

N 

Sg[P] = ^P, fc log = -A; ^ logP, . 

g « 1=1 

Therefore Shannon's quantitative measure of the amount of missing information, 
the entropy of the probabiUty distribution pi , . . . , p„ is 

n 

S\p] = -k Y.Pi log Pi ■ (4.15) 

i=l 

Comments 

Notice that for discrete probability distributions we have Pi <l and log Pi < 0. 
Therefore 5 > for fc > 0. As long as we interpret S as the amount of 
uncertainty or of missing information it cannot be negative. We can also check 
that in cases where there is no uncertainty we get S = 0: if any state has 
probability one, all the other states have probability zero and every term in S 
vanishes. 

The fact that entropy depends on the available information implies that 
there is no such thing as the entropy of a system. The same system may have 
many different entropies. Notice, for example, that already in the third axiom 
we find an explicit reference to two entropies S[p] and Sg[P] referring to two 
different descriptions of the same system. Colloquially, however, one does refer 
to the entropy of a system; in such cases the relevant information available about 
the system should be obvious from the context. In the case of thermodynamics 
what one means by the entropy is the particular entropy that one obtains when 
the only information available is specified by the known values of those few 
variables that specify the thermodynamic macrostate. 

The choice of the constant k is purely a matter of convention. A convenient 
choice is fc = 1. In thermodynamics the choice is Boltzmann's constant ks = 
1.38 X 10~^^erg/K which refiects the historical choice of units of temperature. 
In communication theory and computer science, the conventional choice is A; = 
l/loge2 w 1.4427, so that 

S\p] = - tpi log2 Pi ■ (4.16) 

1=1 

The base of the logarithm is 2, and the entropy is said to measure information 
in units called 'bits'. 

Now we turn to the question of interpretation. Earlier we mentioned that 
from axiom 3 it seems more appropriate to interpret 5 as a measure of the 
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expected rather than the actual amount of missing information. If one adopts 
this interpretation, the actual amount of information that we gain when we find 
that i is the true alternative would have to be log 1 /pi . But this is not quite 
satisfactory. Consider a variable that takes just two values, with probability p 
and 1 with probability 1—p. For very small p, log 1/p would be very large, while 
the information that communicates the true alternative is conveyed by a very 
short one bit message, namely "0" . It appears that it is not the actual amount 
of information that log 1 /p seems to measure but rather how unexpected or how 
surprising the piece of information might be. Accordingly, log 1/pi is sometimes 
called the "surprise" of i. 

It seems reasonable to expect that more information implies less uncertainty. 
We have used the word 'uncertainty' as roughly synonymous to 'lack of infor- 
mation'. The following example illustrates the potential pitfalls. I normally 
keep my keys in my pocket. My state of knowledge about the location of my 
keys is represented by a probability distribution that is sharply peaked at my 
pocket and reflects a small uncertainty. But suppose I check and I flnd that 
my pocket is empty. Then my keys could be virtually anywhere. My new state 
of knowledge is represented by a very broad distribution that reflects a high 
uncertainty. We have here a situation where more information has increased 
the uncertainty rather than decreased it. 

The point of these remarks is not to suggest that there is something wrong 
with the mathematical derivation - eq.(4.15l does follow from the axioms - but 
to suggest caution when interpreting S. The notion of information is at this 
point still vague. Any attempt to find its measure will always be open to the 
objection that it is not clear what it is that is being measured. Indeed, the first 
two of Shannon's axioms seem to be particularly intuitive, but the third one, 
the grouping property, is not nearly as compelling. Is entropy the only way to 
measure uncertainty? Doesn't the variance also measure uncertainty? Shannon 
and Jaynes both argued that one should not place too much significance on the 
axiomatic derivation of eq.(4.15l, that its use can be fully justified a posteriori 
by its formal properties, for example, by the various inequalities it satisfies. 
However, this position can be questioned on the grounds that it is the axioms 
that confer meaning to the entropy; the disagreement is not about the actual 
equations, but about what they mean and, ultimately, about how they should 
be used. Other measures of uncertainty can be introduced and, indeed, they 
have been introduced by Renyi and by Tsallis, creating a whole industry of 
alternative theories. [Renyi 61, Tsallis 88] Whenever one can make an inference 
using Shannon's entropy, one can make other inferences using any one of the 
Renyi's entropies. Which, among all those alternatives, should one choose? 



The two-state case 

To gain intuition about S[p\ consider the case of a variable that can take two 
values. The proverbial example is a biased coin - for example, a bent coin - for 
which the outcome 'heads' is assigned probability p and 'tails' probability 
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The corresponding entropy is 

S{p) = -p log p-{l-p) log (1 - p) , (4.17) 

where we chose fc = 1. It is easy to check that S > and that the maximum 
uncertainty, attained ioi p — 1/2, is S'max = log 2. 

An important set of properties of the entropy follows from the concavity 
of the entropy which follows from the concavity of the logarithm. Suppose 
we can't decide whether the actual probability of heads is pi or p2. We may 
decide to assign probability q to the first alternative and probability 1 — g to 
the second. The actual probability of heads then is the mixture qpi + (1 — q)p2- 
The corresponding entropies satisfy the inequality 

S {qpi + (1 - q)p2) > qS (pi) + {1 - q) S (pa) , (4.18) 



with equality in the extreme cases where pi — p2, or q — 0, or q — 1. Eq.( 4.18"| 



says that however ignorant we might be when we invoke a probability distribu- 
tion, an uncertainty about the probabilities themselves will introduce an even 
higher degree of ignorance. 



4.2 Relative entropy 

The following entropy-like quantity turns out to be useful 

K[p,q] ^+J:p, log ^. (4.19) 

i Qi 

Despite the positive sign K is sometimes read as the 'entropy of p relative to 
q,' and thus called "relative entropy." It is easy to see that in the special case 
when qi is a uniform distribution then K is essentially equivalent to the Shannon 



entropy - they differ by a constant. Indeed, for qt — 1/n, eq. (4.191 becomes 



K[p, 1/n] = (logp, + log n) = log n - S[p] . (4.20) 

i 

The relative entropy is also known by many other names including cross 
entropy, information divergence, information for discrimination, and KuUback- 
Leibler distance [KuUback 59] who recognized its importance for applications in 



statistics, and studied many of its properties). However, the expression (4.191 
has a much older history. It was already used by Gibbs in his Elementary 
Principles of Statistical Mechanics [Gibbs 1902]. 

It is common to interpret K[p, q] as the amount of information that is gained 
(thus the positive sign) when one thought the distribution that applies to a 
random process is q and one learns that the distribution is actually p. The 
interpretation suffers from the same conceptual difficulties mentioned earlier 
concerning the Shannon entropy. In the next chapter we will see that the relative 
entropy turns out to be the fundamental quantity for inference - indeed, more 
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fundamental, more general, and therefore, more useful than entropy itself - and 
that the interpretational difficulties that afflict the Shannon entropy can be 

avoided. (We will also redefine it with a negative sign, ^[p, g] —K[p,q], so 
that it really is a true entropy.) In this chapter we just derive some properties 
and consider some applications. 

An important property of the relative entropy is the Gibbs inequality, 

K[p,q]>0, (4.21) 

with equality if and only if pi = qi for all i. The proof uses the concavity of the 
logarithm, 

logx < a; - 1 or log — < — - 1 , (4.22) 
Pi Pi 



which implies 



Eftlog^ <Efe-P.)-0. (4.23) 

i Pi i 



The Gibbs inequality provides some justification to the common interpreta- 
tion of K[p^ g] as a measure of the "distance" between the distributions p and 
q. Although useful, this language is not quite correct because K[p,q\ ^ K[q,p] 
while a true distance d is required to be symmetric, — d[q,p\. However, 

as we shall later see, if the two distributions are sufficiently close the relative 
entropy K[p + Sp,p] satisfies all the requirements of a metric. Indeed, it turns 
out that up to a constant factor, it is the only natural Riemannian metric on 
the manifold of probability distributions. It is known as the Fisher-Rao metric 
or, perhaps more appropriately, the information metric. 



The two inequalities S[p\ > and K[p,q] > together with eq.(4.20l imply 



0<5'b]<logn, (4.24) 

which establishes the range of the entropy between the two extremes of complete 
certainty (pi = 6ij for some value j) and complete uncertainty (the uniform 
distribution) for a variable that takes n discrete values. 



4.3 Joint entropy, additivity, and subadditivity 

The entropy S[px] reflects the uncertainty or lack of information about the 
variable x when our knowledge about it is codified in the probability distribution 
Px- It is convenient to refer to S[px] directly as the "entropy of the variable x" 
and write 

Sx S[px] = - Ep- log Px . (4.25) 

X 

The virtue of this notation is its compactness but one must keep in mind the 
same symbol x is used to denote both a variable x and its values Xi . To be more 
explicit , 

- J2Px log Px ^ - Y.Px{xi) log Px{xi) ■ (4.26) 

X i 
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The uncertainty or lack of information about two (or more) variables x and 
y is expressed by the joint distribution p^y and the corresponding joint entropy 
is 

Sxy = -Y^PxylogPxy . (4.27) 

xy 

When the variables x and y are independent, p^y — PxPy, the joint entropy 
is additive 

Sxy = - J2PxPy '^OgiPxPy) ^ + Sy , (4.28) 

xy 

that is, the joint entropy of independent variables is the sum of the entropies 
of each variable. This additivity property also holds for the other measure of 
uncertainty we had introduced earlier, namely, the variance, 

var(a; + y) — var(x) + var(y) . (4.29) 

In thermodynamics additivity is called extensivity: the entropy of an ex- 
tended system is the sum of the entropies of its parts provided these parts are 
independent. The thermodynamic entropy can be extensive only when the in- 
teractions between various subsystems are sufficiently weak that correlations 
between them can be neglected. 

When the two variables x and y are not independent the equality (4.281 
can be generalized into an inequality. Consider the joint distribution p^y = 
PxPy\x — PyPx\y The relative entropy or KuUback "distance" of p^y to the 
product distribution PxPy that would represent uncorrelated variables is given 
by 

K[poi:y,PxPy] = J2Pxy log 

xy PxPy 

= -Sxy ~ Y^Pxy logpx - Y^Pxy ^OgPy 

xy xy 

= -Sxy + Sx + Sy . (4.30) 
Therefore, using X > we get 

Sxy <Sx + Sy, (4.31) 

with the equality holding when the two variables x and y are independent. 
This inequality is called the subadditivity property. Its interpretation is clear: 
entropy increases when information about correlations is discarded. 



4.4 Conditional entropy and mutual information 

Consider again two variables x and y. We want to measure the amount of 
uncertainty about one variable x when we have some limited information about 
another variable y. This quantity, called the conditional entropy, and denoted 
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S^\y, is obtained by calculating the entropy of a; as if the precise value of y were 
known and then taking the expectation over the possible values of y 

Sx\y = J2PyS[Px\y] = - J2PvJ2Px\y log Px\y = ~T.Pxy log Px\y , (4.32) 

y y X x,y 

where Pxy is the joint distribution of x and y. 

The conditional entropy is related to the entropy of x and the joint entropy 
by the following "chain rule." Use the product rule for the joint distribution 

log Pxy = log Py + log p^iy , (4.33) 

and take the expectation over x and y to get 

Sxy — Sy + Sx\y ■ (4.34) 

In words: the entropy of two variables is the entropy of one plus the conditional 
entropy of the other. Also, since Sy is positive we see that conditioning reduces 
entropy, 

Sxy > S,\y . (4.35) 

Another useful entropy-like quantity is the so-called "mutual information" 
of a; and y, denoted M^y, which "measures" how much information x and y have 
in common. This is given by the relative entropy between the joint distribution 
Pxy and the product distribution PxPy that discards all information contained 
in the correlations. Using eq.(4.30), 

Mxy =^ K[pxy,PxPy] ^ Sx + Sy -Sxy>0, (4.36) 



which shows that it is symmetrical in x and y. Using eq.(4.34l the mutual 
information is related to the conditional entropies by 

Mxy ^ Sx ^ Sx\y — Sy — Sy^x ■ (4.37) 

The relationships among these various entropies can be visualized by a figure 
that resembles a Venn diagram. (The diagram is usually considered a purely 
mnemonic aid, but recent work [Knuth 02-06] on the duality between asser- 
tions and questions, and the corresponding duality between probabilities and 
entropies suggests that the resemblance between the two types of Venn diagrams 
is not accidental.) 



4.5 Continuous distributions 



Shannon's derivation of the expression for entropy, eq.(4.15 1, applies to probabil- 
ity distributions of discrete variables. The generalization to continuous variables 
is not quite straightforward. 

The discussion will be carried out for a one-dimensional continuous variable; 
the generalization to more dimensions is trivial. The starting point is to note 
that the expression 

- J dxp{x) log p{x) (4.38) 
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is unsatisfactory. A change of variables x ^ y = y{x) changes the probabil- 
ity density p{x) to p'{y) but does not represent a loss or gain of information. 
Therefore, the actual probabilities do not change, p{x)dx — p'{y)dy, and neither 
should the entropy. However, one can check that (4.38) is not invariant. 



Jdxp{x) log p{x) = Jdyp'{y) log 



dy^ 
dx 



^ I dy p'{y) log p'{y) 



(4.39) 



We approach the continuous case as a limit from the discrete case. Consider 
a continuous distribution p{x) defined on an interval for Xa < x < Xb- Divide 
the interval into equal intervals Ax — (xb — Xa) /N. The distribution p{x) can 
be approximated by a discrete distribution 



Pn = p{xn)Ax , 

where x„ — Xa + nAx and n is an integer. The discrete entropy is 



N 

Sn = -J2 ^xp{xn) log [p{xn)Ax\ , 

n=0 



and as iV — > CX3 we get 



Sn — > log — / dxp{x) log 



p{x) 



1/ {xb - Xa) 



(4.40) 



(4.41) 



(4.42) 



which diverges. This is quite to be expected: it takes a finite amount of informa- 
tion to identify one discrete alternative within a finite set, but it takes an infinite 
amount to single out one point in a continuum. The difference Sn ~ log N has 
a well defined limit and we are tempted to consider 



/ dxp{x) log 



p{x) 



1/ {xb - Xa) 



(4.43) 



as a candidate for the continuous entropy, until we realize that, except for 



an additive constant, it coincides with the unacceptable expression (4.381 and 



should be discarded for precisely the same reason: it is not invariant under 
changes of variables. Had we first changed variables to y = y{x) and then 
discretized into N equal Ay intervals we would have obtained a different limit 



Vb 

' Jdyp'iy) log 

Va 



P'iv) 



1/ {Vb ~ Va) 



(4.44) 



The problem is that the limiting procedure depends on the particular choice of 
discretization; the limit depends on which particular set of intervals Aa; or Ay 
we have arbitrarily decided to call equal. Another way to express the same idea 
is to note that the denominator 1/ [xb — Xa) in (4.43) represents a probability 
density that is uniform in the variable x, but not in y. Similarly, the density 
1/ {yb — J/a) in (4.44) is uniform in y, but not in x. 
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Having identified the origin of the problem we can now suggest a solution. 
On the basis of our prior knowledge of the problem at hand we must decide on 
a privileged set of equal intervals, or alternatively, on one preferred probability 
distribution we are willing to define as "uniform." Then, and only then, it 
makes sense to propose the following definition 

S[p, fi] - Jdxp{x) log ^ . (4.45) 

It is easy to check that this is invariant, 

Jdxpix) log ^ = Jdyp'iy) log ^ . (4.46) 

Examples illustrating possible choices of the uniform /i(x) are the following. 

1. When the variable x refers to position in "physical" space, we can feel 
fairly comfortable with what we mean by equal volumes: use Cartesian 
coordinates and choose fi{x) = constant. 

2. In a curved space I?-dimensional, with a known metric tensor gij, i.e., the 
distance between neighboring points with coordinates x'' and x^ + dx^ is 
given by df."^ = gijdx^dx^ , the volume elements are given by (detg)^^^ d^x. 
In this case choose ^(x) oc {detg)^^^. 

3. In classical statistical mechanics the Hamiltonian evolution in phase space 
is, according to Liouville's theorem, such that phase space volumes are 
conserved. This leads to a natural definition of equal intervals or equal 
volumes. The corresponding choice of uniform /x is called the postulate of 
"equal a priori probabilities." 



Notice that the expression in eq.(4.45) is a relative entropy —K[p,fi\. This 



is a hint for a theme that will be fully developed in chapter 6: relative entropy 
is the more fundamental quantity. Strictly, there is no Shannon entropy in the 
continuum - not only do we have to subtract an infinite constant and spoil 
its (already shaky) interpretation as an information measure, but we have to 
appeal to prior knowledge and introduce the measure /i. On the other hand 
there are no difficulties in obtaining the continuum relative entropy from its 
discrete version. We can check that 

i^A. - E log — = £ Axp(x„) log (4.47) 

n=0 Qn n=0 q(Xn)^X 



has a well defined limit, 



K[p,q] ^ J dx p{x) log^ , (4.48) 



which is explicitly invariant under coordinate transformations. 
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4.6 Communication Theory 

Here we give the briefest introduction to some basic notions of communication 
theory as originally developed by Shannon [Shannon 48, Shannon Weaver 49]. 
For a more comprehensive treatment see [Cover Thomas 91]. 

Communication theory studies the problem of how a message that was se- 
lected at some point of origin can be best reproduced at some later destination 
point. The complete communication system includes an information source that 
generates a message composed of, say, words in English, or pixels on a picture. 
A transmitter translates the message into an appropriate signal. For example, 
sound pressure is encoded into an electrical current, or letters into a sequence 
of zeros and ones. The signal is such that it can be transmitted over a commu- 
nication channel, which could be electrical signals propagating in coaxial cables 
or radio waves through the atmosphere. Finally, a receiver reconstructs the sig- 
nal back into a message that can be interpreted by an agent at the destination 
point. 

From the engineering point of view the communication system must be de- 
signed with only a limited information about the set of possible messages. In 
particular, it is not known which specific messages will be selected for transmis- 
sion. The typical sort of questions one wishes to address concern the minimal 
physical requirements needed to communicate the messages that could poten- 
tially be generated by a particular information source. One wants to characterize 
the sources, measure the capacity of the communication channels, and learn how 
to control the degrading effects of noise. And after all this, it is somewhat ironic 
but nevertheless true that "information theory" is completely unconcerned with 
whether any "information" is being communicated at all. As far as the engi- 
neering goes, whether the messages convey some meaning or not is completely 
irrelevant. 

To illustrate the basic ideas consider the problem of "data compression." A 
useful idealized model of an information source is a sequence of random variables 
xi,X2, ■ ■ ■ which take values from a finite alphabet of symbols. We will assume 
that the variables are independent and identically distributed. (Eliminating 
these limitations is both possible and important.) Suppose that we deal with a 
binary source in which the variables Xj, which are usually called 'bits', take the 
values zero or one with probabilities p or 1— p respectively. Shannon's idea was 
to classify the possible sequences xi, . . . ,xiy into typical and atypical according 
to whether they have high or low probability. For large A'^ the expected number 
of zeros and ones is Np and N{1 — p) respectively. The probability of these 
typical sequences is 

P(xi,...,xw)«p^''(l-p)^(i-f) , (4.49) 

so that 

- log P{xu- . . , xjv) « -N[plogp -{1-p) log(l - p)] = NSip) (4.50) 



where S{p) is the two-state entropy, eq.(4.17), the maximum value of which is 
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Sniax — log 2. Therefore, the probability of typical sequences is roughly 

P(xi,...,a;^)«e-^^(f). (4.51) 

Since the total probability is less than one, we see that the number of typical 
sequences has to be less than about e^'^^^-' which for large N is considerably 
less than the total number of possible sequences, 2^ = e^'°s2^ This fact is 
very significant. Transmitting an arbitrary sequence irrespective of whether it 
is typical or not requires a long message of N bits, but we do not have to waste 
resources in order to transmit all sequences. We only need to worry about the 
far fewer typical sequences because the atypical sequences are too rare. The 
number of typical sequences is about 

gAfS(p) ^ 2^'5(p)/i°g2 ^ 2^'^(^')/'^""'- (4.52) 

and therefore we only need about NS{p) /S'max bits to identify each one of 
them. Thus, it must be possible to compress the original long message into 
a much shorter one. The compression might imply some small probability of 
error because the actual message might conceivably turn out to be atypical but 
one can, if desired, avoid any such errors by using one additional bit to flag 
the sequence that follows as typical and short or as atypical and long. Actual 
schemes for implementing the data compression are discussed in [Cover Thomas 
91]. 

Next we state these intuitive notions in a mathematically precise way. 
Theorem: The Asymptotic Equipartition Property (AEP). If xi,...,X]y are 
independent variables with the same probability distribution p{x), then 

— log P{xi, . . . ,xn) — > S[p] in probability. (4.53) 

Proof: If the variables Xi are independent, so are their logarithms, logp{xi), 

1 1 ^ 

- — logF(a;i, ...,xn) = ~j^J2^ogp{xt) , (4.54) 

and the law of large numbers (see section 2.8) gives 

< £ 



lim Prob 

N^co 



~logP{xi,. . . ,xn) + {logp{x)) 



(4.55) 



where 

- (logpix)) = S[p] . (4.56) 

This concludes the proof. 

We can elaborate on the AEP idea further. The typical sequences are those 



for which eq.(4.51 1 or (4.531 is satisfied. To be precise let us define the typical 



set Atv,^ as the set of sequences with probability P{xi, . . . , zjy) such that 

-Nisip)+e] < p(^^^ ...,xn)< e-^[^(P)-^l . (4.57) 



e 
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Theorem of typical sequences: 

(1) For N sufficiently large Prob[^Ar^e] > 1 — e. 

(2) l^iv.el < e^['^(p)+'^l where |Ajv.j| is the number of sequences in Ajv,£. 

(3) For N sufficiently large \AN,e\ > (1 - e)e^[^(P)-^l. 

In words: the typical set has probability near one, typical sequences are nearly 
equally probable (thus the 'equipartition'), and there are about e^^^P^ of them. 
To summarize: 

Almost all events are almost equally likely. 



Proof: Eq.(4.55l states that for fixed e, for any given S there is an iV^ such 

ave 

logP(a;i, . . .,xn) + S[p] 



that for all N > Ns, we have 
1 



Prob 



N' 



< e 



>l-5. (4.58) 



Thus, the probability that the sequence {xi, . . . ,xm) is e- typical tends to one, 
and therefore so must Prob[ylAr.j]. Setting 6 = e yields part (1). To prove (2) 
write 

1 > Prob[AAr,,] = P{xi,...,xn) 

(xi,...,a:jv)eAjv_e 

> ^ ^'N[S[p)+e\ ^ ^-N[Sip)+e\ |^^_^| (4 59) 

(xi,...,Xn)^An,s 

Finally, from part (1), 

1 - e < Prob[^Ar,,] = Pixi,...,XN) 

{xi,...,X]v)eAN,e 

< y: e-^[«(^')'^l=.e-^[^(f)-^l|A^,,| , (4.60) 

{xi,...,Xn)&An^^ 

which proves (3). 

We can now quantify the extent to which messages generated by an infor- 
mation source of entropy S[p\ can be compressed. A scheme that produces 
compressed sequences that are more than NS{p) / Sma,x bits is capable of dis- 
tinguishing among all the typical sequences. The compressed sequences can be 
reliably decompressed into the original message. Conversely, schemes that yield 
compressed sequences of fewer than NS{p) / 5'max bits cannot describe all typi- 
cal sequences and are not reliable. This result is known as Shannon's noiseless 
channel coding theorem. 
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4.7 Assigning probabilities: MaxEnt 

Probabilities are introduced to deal with lack of information. The notion that 
entropy S\p] can be interpreted as a quantitative measure of the amount of 
missing information has one remarkable consequence: it provides us with a 
method to assign probabilities. The idea is simple: 

Among all possible probability distributions that agree with whatever we know 
select that particular distribution that reflects maximum ignorance about every- 
thing else. Since ignorance is measured by entropy, the method is mathematically 
implemented by selecting the distribution that maximizes entropy subject to the 
constraints imposed by the available information. This method of reasoning is 
called the method of Maximum Entropy, and is often abbreviated as MaxEnt. 

Ultimately, the method of maximum entropy is based on an ethical principle 
of intellectual honesty that demands that one should not assume information 
one does not have. The idea is quite compelling but its justification relies heavily 
on interpreting entropy as a measure of missing information and therein lies its 
weakness: to what extent are we sure that entropy is the unique measure of 
information or of uncertainty? 

As a simple illustration of the MaxEnt method in action consider a variable 
X about which absolutely nothing is known except that it can take n discrete val- 
ues Xi with i = 1 . . .n. The distribution that represents the state of maximum 
ignorance is that which maximizes the entropy subject to the single constraint 
that the probabilities be normalized, ^^Pi = 1. Introducing a Lagrange multi- 
plier a to handle the constraint, the variation Pi Pi + Spi gives 



= 5 ^S[p] - aj:p,j = - X: (log Pr + l + a) 6p, , (4.61) 
so that the selected distribution is 

p, = e-i-" or = - , (4.62) 
n 

where the multiplier a has been determined from the normalization constraint. 
We can check that the maximum value attained by the entropy, 

S'niax = ~J2i - log - = log , (4.63) 
n n 



agrees with eq.(4.24l 



Remark: The distribution of maximum ignorance turns out to be uniform. 
It coincides with what we would have obtained using Laplace's Principle of 
Insufficient Reason. It is sometimes asserted that the MaxEnt method provides 
a proof of Laplace's principle but such a claim is questionable. As we saw 
earlier, the privileged status of the uniform distribution was imposed through 
the Shannon's axioms from the very beginning. 
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4.8 Canonical distributions 

The available information constrains the possible probability distributions. Al- 
though the constraints can take any form whatsoever, in this section we develop 
the MaxEnt formalism for the special case of constraints that are linear in the 
probabilities. The most important applications are to situations of thermo- 
dynamic equilibrium where the relevant information is given in terms of the 
expected values of those few macroscopic variables such as energy, volume, and 
number of particles over which one has some experimental control. (In the next 
chapter we revisit this problem more explicitly.) 

The goal is to select the distribution of maximum entropy from within the 
family of all distributions for which the expectations of some functions f^{x), 
k = 1,2,... have known numerical values , 



if) = EP^fl: = F'^ , (4.64) 

i 

where we set f^{xi) = to simplify the notation. In addition there is a 
normalization constraint, = 1. Introducing the necessary multipliers, the 
entropy maximization is achieved setting 

= 5 (^S[p] - aY^p, - Afe (/'=) 
= -E(logP. + l + «+Afe/f)<5p, , (4.65) 

i 

where we adopt the summation convention that repeated upper and lower indices 
are summed over. The solution is the so-called 'canonical' distribution, 

= exp-(Ao-HAfc/f) , (4.66) 

where we have set 1 + a = Aq. The normalization constraint determines Ag, 

= Eexp(-Afe/f ) = Z (Ai, A2, . . .) (4.67) 



where we have introduced the partition function Z. Substituting eqs.(4.66l 



and (4.671 into the other constraints, eqs.(4.64l, gives a set of equations that 



implicitly determine the remaining multipliers 

dlogZ 



, (4.68) 



and substituting into S[p\ = —"^Pi log pi we obtain the maximized value of the 
entropy, 

^max = Ep.(Ao + \kft) - Ao + XkF'' . (4.69) 



Equations ( |4.66|4.68 l are a generalized form of the "canonical" distributions 



first discovered by Maxwell, Boltzmann and Gibbs. Strictly, the calculation 
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above only shows that the entropy is stationary, 5S = 0. To complete the 



argument we must show that (4.691 is the absolute maximum rather than just 



a local extremum or a stationary point. 

Consider any other distribution qi that satisfies precisely the same con- 
straints in eqs.(4.64). According to the basic Gibbs inequality for the relative 



entropy of q and the canonical p, 

K{q,p) 



log - > 

Pi 



or 



s[q] < -^J2n log Pi 



Substituting eq.(4.66) gives 



S[q] < E%(Ao + Afc/f ) = Ao + XkF'' 



(4.70) 



(4.71) 



(4.72) 



Therefore 

S[q] < S\p] = ^„,ax . (4.73) 
In words: within the family of all distributions q that satisfy the constraints 



(4.641 the distribution that achieves the maximum entropy is the canonical 



distribution p given in eq.(4.66) 



Having found the maximum entropy distribution we can now develop the 
MaxEnt formalism along lines that closely parallel the formalism of statistical 
mechanics. Each distribution within the family (4.661 can be thought of as a 
point in a continuous space - the manifold of canonical distributions. Each 
specific choice of expected values {F^ , F"^ , . . .) determines a unique point within 
the space, and therefore the F^ play the role of coordinates. To each point 
{F^,F^, . . .) we can associate a number, the value of the maximized entropy. 
Therefore, iSmax is a scalar field which we denote S{F^ , F'^ , . . .) — S{F). In 
thermodynamics it is conventional to drop the suffix 'max' and to refer to S{F) 
as the entropy of the system. This language is inappropriate because it can 
be misleading. We should constantly remind ourselves that S{F) is just one 
out of many possible entropies. S{F) is that particular entropy that measures 
the amount of missing information of a subject whose knowledge consists of the 
numerical values of the _Fs and nothing else. The multiplier 



Ao = logZ(Ai,A2,...) = logZ(A) 



(4.74) 



is sometimes called the "free energy" because it is closely related to the ther- 
modynamic free energy. 



SiF)^logZiX) 



(4.75) 



The quantities S{F) and log Z{X) contain the same information; the equation 
above shows that they are Legendre transforms of each other. Just as the Fs 



are obtained from logZ(A) from eq.(4.68l, the As can be obtained from S{F) 
dS{F) _ aiogZ(A) dXj 



dX, 



dX^ 

Qpk ^ Qpk 



F' + Xk 



(4.76) 
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or, using eq.(4.68), 



dS{F) 



(4.77) 



which shows that the multiphers are the components of the gradient of the 
entropy S{F) on the manifold of canonical distributions. 

A useful extension of the formalism is the following. It is common that the 
functions f'' are not fixed but depend on one or more parameters v that can 
be externally manipulated, = f''{xi,v). For example /f could refer to the 
energy of the zth state of the system, and the parameter v could be the volume 
of the system or an externally applied magnetic field. 

Then a general change in the expected value F*^ induced by changes in both 
f'' and A/c, is expressed as 



i 

The first term on the right is 

Sv 



^ ' j ov 



dv 



(4.78) 



(4.79) 



When represents the internal energy then (^5f^) is a small energy trans- 
fer that can be controlled through an external parameter v. This suggests 
that {Sf'') represents a kind of "generalized work," 5W'' , and the expectations 
{df^/dv) are analogues of pressure or susceptibility. 



dv 



Sv . 



The second term in eq.(4.78). 



SQ' 



dcf 



(4.80) 



(4.81) 



is a kind of "generalized heat" , and 

SF'' = SW'' 



(4.82) 



is a "generalized first law." 

The corresponding change in the entropy is obtained from eq.(4.75l. 



SS^SlogZ{X)+S{XkF'') 



1 



fel ^-^kf^ 



SXkF'' 



XkSF'^ 



= x, is{f)-{sf)), 

which, using eq.( 4.81[ ), gives 



(4.83) 
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5S = XkSQ'' 



(4.84) 



It is easy to see that this is equivalent to eq.(4.77l where the partial derivatives 
are derivatives at constant v. 

Thus the entropy remains constant in infinitesimal "adiabatic" processes - 
those with SQ'' = 0. From the information theory point of view [see eq.(4.81 1] 
this result is a triviality: the amount of information in a distribution cannot 
change when the probabilities do not change, 



Sp^ = 



= O^SS = 



(4.85) 



4.9 On constraints and relevant information 

The method of maximum entropy has been successful in many applications, but 
there are cases where it has failed. Are these symptoms of irreparable flaws or 
mere examples of misuses of the method? MaxEnt is a method for processing 
information: what information are we talking about? The importance of this 
issue cannot be overestimated. Here we collect a few remarks; this is a topic to 
which we will return repeatedly. 

One point that must be made is that questions about how information is 
processed ~ and this is the problem that MaxEnt is supposed to address - 
should not be confused with questions about how the information was obtained 
in the first place. These arc two separate issues. 

Here is an example of a common error. Once we accept that certain con- 
straints might refer to the expected values of certain variables, how do we decide 
their numerical magnitudes? The numerical values of expectations are seldom 
known and one might be tempted to replace expected values by sample averages 
because it is the latter that are directly available from experiment. But the two 
are not the same: Sample averages are experimental data. Expected values are 
not experimental data. 

For very large samples such a replacement can be justified by the law of large 
numbers - there is a high probability that sample averages will approximate the 
expected values. However, for small samples using one as an approximation for 
the other can lead to incorrect inferences. It is important to realize that these 
incorrect inferences do not represent an intrinsic flaw of the MaxEnt method; 
they are merely a warning of how the MaxEnt method should not be used. 

There are many other objections that have been raised against the logic 
behind the MaxEnt method. We make no attempt to survey them all; many 
have already received adequate answers (see, e.g., [Jaynes 83] and [Jaynes 03], 
particularly section 11.8). But some objections remain that are quite legitimate 
and demand our attention. They revolve around the following question: Once 
we accept that constraints will be in the form of the expected values of certain 
variables, how do we decide which variables to choose? 
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When using the MaxEnt method to obtain, say, the canonical Boltzniann 
distribution {pi oc e~^^^) it has been common to adopt the following language: 

(A) We seek the probability distribution that codifies the information we ac- 
tually have (say, the expected energy) and is maximally unbiased (i.e. 
maximally ignorant or maximum entropy) about all the other information 
we do not possess. 

Many authors find this justification unsatisfactory. Indeed, they might argue, 
for example, that 

(Bl) The observed spectrum of black body radiation is whatever it is, inde- 
pendently of the information that happens to be available to us. 

We prefer to phrase the objection differently: 

(B2) In most realistic situations the expected value of the energy is not a 
quantity we happen to know; how, then, can we justify using it as a 
constraint? 

Alternatively, even when the expected values of some quantities happen to be 
known, according to (A) what MaxEnt provides is the best possible inferences 
given the limited information that is available. This is no mean feat, but there 
is no guarantee that the resulting inferences will be any good at all. The pre- 
dictions of statistical mechanics are spectacularly accurate: how can we hope 
to achieve equally spectacular predictions in other fields? 

(B3) We need some understanding of which are the "correct" quantities the 
expectation values of which codify the relevant information for the problem 
at hand. 

Merely that some particular expected value happens to be known is neither an 
adequate nor a sufficient explanation. 

A partial answer to these objections starts with the observation that whether 
the value of the expected energy is known or not, it is nevertheless still true that 
maximizing entropy subject to the energy constraint leads to the indisputably 
correct family of thermal equilibrium distributions (e.g., the black-body spectral 
distribution). The justification behind imposing a constraint on the expected 
energy cannot be that this is a quantity that happens to be known - because 
of the brute fact that it is not known - but rather that the expected energy 
is the quantity that should be known. Even when its actual numerical value 
is unknown, we recognize it as the relevant information without which no suc- 
cessful predictions are possible. (In the next chapter we revisit this important 
question.) 

Therefore we allow MaxEnt to proceed as if this crucial information were 
available which leads us to a family of distributions containing the temperature 
as a free parameter. The actual value of this parameter will have to be inferred 
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from the experiment itself either directly, using a thermometer, or indirectly by 
Bayesian analysis from other empirical data. 

To summarize: It is not just what you happen to know; you have to know 
the right thing. The constraints that should be imposed are those that codify 
the information that is relevant to the problem under consideration. Between 
one extreme of ignorance (we know neither which variables are relevant nor 
their expected values), and the other extreme of useful knowledge (we know 
which variables are relevant and we also know their expected values), there 
is an intermediate state of knowledge - and this is the rule rather than the 
exception - in which the relevant variables have been correctly identified but 
their actual expected values remain unknown. In this intermediate state, the 
information about which are the relevant variables is taken into account using 
MaxEnt to select a parametrized family of probability distributions, while the 
actual expected values must then be inferred independently either by direct 
measurement or inferred indirectly using Bayes' rule from other experimental 
data. 

Achieving this 'intermediate state of knowledge' is the difficult problem pre- 
sented by (B3). Historically progress has been achieved in individual cases 
mostly by "intuition," that is, trial and error. Perhaps the seeds for a more 
systematic "theory of relevance" can already be seen in the statistical theories 
of model selection and of non-parametric density estimation. 



Chapter 5 

Statistical Mechanics 



Among the various theories that make up what we call physics, thermodynamics 
holds a very special place because it provided the first example of a fundamental 
theory that could be interpreted as a procedure for processing relevant infor- 
mation. Our goal in this chapter is to provide a more explicit discussion of 
statistical mechanics as a theory of inference. We show that several notoriously 
controversial topics such as the Second Law of thermodynamics, irreversibil- 
ity, reproducibility, and the Gibbs paradox can be considerably clarified when 
viewed from the information/inference perspective. 

Since the success of any problem of inference hinges on identifying the rele- 
vant information we start by providing some background on the dynamical evo- 
lution of probability distributions - Liouville's theorem - and then we justify 
why in situations of thermal equilibrium the relevant constraint is encapsulated 
into the expected value of the energy (and/or other such conserved quantities). 

5.1 Liouville's theorem 

Perhaps the most relevant, and therefore, most important piece of information 
that has to be incorporated into any inference about physical systems is that 
their time evolution is constrained by equations of motion. Whether these equa- 
tions - those of Newton, Maxwell, Yang and Mills, or Eiiisteiu can themselves 
be derived as examples of inference are questions which will not concern us at 
this point. 

To be specific, in this section we will limit ourselves to disc 'lis SI 11^ C'li-issiCiril 
systems such as fluids. In this case there is an additional crucial piece of relevant 
information: these systems are composed of molecules. For simplicity we will 
assume that the molecules have no internal structure, that they are described 
by their positions and momenta, and that they behave according to classical 
mechanics. 

The import of these remarks is that the proper description of the mAcrostate 
of a fluid of N particles in a volume V is in terms of a "vector" in the A'^-particle 
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phase space, z = (xi,pi, . . . xn,Pn)- The time evolution is given by Hamilton's 
equations, 

— - — and — - -— (5 1) 

dt dpi dt dxi ' 

where H is the Hamiltonian, 



AT 2 

H= Y.^ + U{x^,...XN,V) . 



(5.2) 



But the actual positions and momenta of the molecules are unknown and thus 
the macrostate of the fluid is described by a probability density in phase space, 
f{z,t). When the system evolves continuously according to Hamilton's equa- 
tions there is no information loss and the probability flow satisfies a local con- 
servation equation, 

-f{z,t)^~V,-J{z,t) , (5.3) 
where the probability current J is a vector given by 



J{z,t) = f{z,t)z^{f{z,t)^,f{z,t)'^P' 



dt 



dt 



(5.4) 



Evaluating the divergence explicitly using (5.11 gives 

dx 



dl 
dt 



N 

i=l 
N 



d_ 

dxi y ' ' dt 

df dH df dH 

'Pi dpi dxi 



d_ 
dpi 



f{z,t) 



dpt 
dt 



(5.5) 



Thus the time derivative of f{z, t) at a fixed point z is given by the Poisson 
bracket with the Hamiltonian H, 



df ^^dg^fdH df dH df 



dt 



ri \dxi dpi 



dxi 



(5.6) 



This is called the Liouville equation. 

Two important corollaries are the following. Instead of focusing on the 
change in f{z,t) at a fixed point z we can study the change in / {z{t),t) at a 
point z{t) that is being carried along by the flow. This defines the so-called 
"convective" time derivative, 



. . . ,^ ,^ d ^ / df dx, df dp. 



(5.7) 



Using Hamilton's equations shows that the second term is —{H, /} and cancels 
the first, therefore 

±f{z{t),t)^0, (5.8) 



5.2. DERIVATION OF EQUAL A PRIORI PROBABILITIES 
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which means that / is constant along a flow line. Explicitly, 

/(z(t),t)-/(z(i'),i') . (5.9) 

Next consider a small volume element Az(t) that is being carried along by 
the fluid flow. Since trajectories cannot cross each other (because Hamilton's 
equations are flrst order in time) they cannot cross the boundary of the evolving 
volume Az{t) and therefore the total probability within Az(t) is conserved, 

|prob[Az(t)] = j^[Az{t)f{z{t),t)] = . (5.10) 



But / itself is constant, eq.(5.8l, therefore 

j^Az{t) = , (5.11) 

which means that the shape of a region of phase space may get deformed by 
time evolution but its volume remains invariant. 



5.2 Derivation of Equal a Priori Probabilities 

Earlier, in section 4.5, we pointed out that a proper definition of entropy in a 



continuum, eq.(4.45 1, requires that one specify a privileged background measure 

S[f,f,]^-Jdzf{z)logl^ , (5.12) 

where dz = d^^xd^^p. The choice of //(z) is important: it determines what we 
mean by a uniform or maximally ignorant distribution. 

It is customary to set /z(z) equal to a constant which we might as well 
choose to be one. This amounts to postulating that equal volumes of phase 
space are assigned the same a priori probabilities. Ever since the introduction of 
Boltzmann's ergodic hypothesis there have been many failed attempts to derive 
it from purely dynamical considerations. In this section we want to determine 
fi{z) by proving the following theorem 

The Equal a Priori Probability Theorem: Since Hamiltonian dynamics 
involves no loss of information, if the entropy S[f, /x] is to be interpreted as the 
measure of amount of information, then fj,{z) must be a constant in phase space. 
Remark: In chapter 6 the requirement that the entropy S must be interpreted 
as a measure of information will be removed and thus the logic of statistical 
mechanics as a theory of inference will be considerably strengthened. 
Proof: The main non-dynamical hypothesis is that entropy measures informa- 
tion. The information entropy of the time-evolved distribution f{z,t) is 

S{t) = -Jdzf{z,t)loglj^ . (5.13) 
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The first input from Hamiltonian dynamics is that information is not lost and 
therefore we must require that S(t) be constant, 



dt 



S{t) = 



Therefore, 



dt 



Sit) = ~Jdz 



df{z,t) ^^^f{z,t) 



dt 



df{z,t) 
dt 



The second term vanishes, 



r, df{z,t) d 
Jdz^^^-Jdzf{z,t)^0 



(5.14) 



(5.15) 



(5.16) 



A second input from Hamiltonian dynamics is that probabilities are not merely 
conserved, they are locally conserved, which is expressed by eqs.(5.3l and (5.4 1. 

f{z,t) 



The first term of eq. (5.151 can be rewritten. 



-5(t) = /d2V, • J(z,t)log ^^^^^ , 
so that integrating by parts (the surface term vanishes) gives 

= Jdz [-Z ■ V,/(z, t) + f{z, t)z ■ V, log ^i{z) 



(5.17) 



(5.18) 



ville's equation (5.6 1, 



Hamiltonian dynamics enters here once again: the first term vanishes by Liou- 

(5.19) 



/dz z ■ VJ{z, t) = Jdz {H, f{z, t)} = Jdz Mfl^ ^ 



dt 



and therefore, imposing ( |5l4| 
d 



dt 



S{t) ^ Jdz f{z, t)z ■ log ^J.{z) ^ 



(5.20) 



This integral must vanish for any arbitrary choice of the distribution f{z,t), 
therefore 

i • V2logAi(z) = . (5.21) 

Furthermore, we have considerable freedom about the particular Hamiltonian 
operating on the system. We could choose to change the volume in any arbi- 
trarily prescribed way by pushing on a piston, or we could choose to vary an 
external magnetic field. Either way we can change H(t) and therefore i at will. 
The time derivative dS/ dt must still vanish irrespective of the particular choice 
of the vector z. We conclude that 



\7z log/x(z) = or ^{z) = const . 



(5.22) 



5.3. THE RELEVANT CONSTRAINTS 
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To summarize: the requirement that information is not lost in Hamiltonian 
dynamics implies that the measure of information must be a constant of the 
motion, 

p{t)^0, (5.23) 
and this singles out the Gibbs entropy, 

S{t)^-Jdzf{z,t)logf{z,t) , (5.24) 
as the correct information entropy. 



It is sometimes asserted that (5.231 implies that the Gibbs entropy cannot 



be identified with the thermodynamic entropy because this would be in contra- 
diction to the second law. As we shall see below, this is not true; in fact, it is 
quite the opposite. 



5.3 The relevant constraints 

Thermodynamics is concerned with situations of thermal equilibrium. What is 
the relevant information needed to make inferences that apply to these special 
cases? The first condition we must impose on / (z, t) to describe equilibrium is 
that it be independent of time. Thus we require that {H, /} = and / must be a 
function of conserved quantities such as energy, momentum, angular momentum, 
or number of particles. But we do not want / to be merely stationary, as say, 
for a rotating fluid, we want it to be truly static. We want / to be invariant 
under time reversal. For these problems it turns out that it is not necessary to 
impose that the total momentum and total angular momentum vanish; these 
constraints will turn out to be satisfied automatically. To simplify the situation 
even more we will only consider problems where the number of particles is held 
fixed. Processes where particles are exchanged as in the equilibrium between a 
liquid and its vapor, or where particles are created and destroyed as in chemical 
reactions, constitute an important but straightforward extension of the theory. 

It thus appears that it is sufficient to impose that / be some function of the 
energy. According to the formalism developed in section 4.8 and the remarks in 
4.9 this is easily accomplished: the constraints codifying the information that 
could be relevant to problems of thermal equilibrium should be the expected 
values of functions (piE) of the energy. For example, {4>{E)) could include 
various moments, (E), {E'^),. . . or perhaps more perhaps complicated functions. 
The remaining question is which functions 4>{E) and how many of them. 

To answer this question we look at thermal equilibrium from the point of view 
leading to what is known as the microcanonical formalism. Let us enlarge our 
description to include the system of interest A and its environment, that is, the 
thermal bath B with which it is in equilibrium. The advantage of this broader 
view is that the composite system C ^ A + B can be assumed to be isolated 
and we know that its energy Ec is some fixed constant. This is highly relevant 
information: when the value of Ec is known, not only do we know (Ec) but we 
know the expected values {(j>{Ec)) for absolutely all functions 4){Ec). In other 
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words, in this case we have succeeded in identifying the relevant information and 
we are finally ready to assign probabilities using the MaxEnt method. (When 
the value of Ec is not known we are in that state of "intermediate" knowledge 
described in section 4.9.) 

To simplify the notation it is convenient to divide phase space into discrete 
cells of equal volume. For system A let the (discretized) microstate Za have 
energy Ea- For the thermal bath B a much less detailed description is sufficient. 
Let the number of bath microstates with energy Eh be flsiEi,). Our relevant 
information includes the fact that A and B interact very weakly, just barely 
enough to attain equilibrium, and thus the known total energy E,. constrains 
the allowed microstates of A + B to the subset that satisfies 

Ea + Eb^Ec. (5.25) 

The total number of such microstates is 

^iEc) ^ Ea^BiEc - Ea) . (5.26) 

We are in a situation where we know absolutely nothing beyond the fact that 
the composite system C can be in any one of its il{Ec) allowed microstates. 
This is precisely the problem tackled in section 4.7: the maximum entropy 



distribution is uniform, eq.(4.62), and the probability of any microstate of C is 
l/ri(£'c). More importantly, the probability that system A is in the particular 
microstate a when it is in thermal equilibrium with the bath B is 

This is the result we sought; now we need to interpret it. It is convenient to 
rewrite pa in terms of the the entropy of the bath Sb — fclog fls, 

Pa^exp^SsiEc- Ea) . (5.28) 

There is one final piece of relevant information we can use: the thermal bath B 
is much larger than system A, Ec 3> Ea, and we can Taylor expand 

SB{Ec--Ea)^SB{Ec)-^ + ... , (5.29) 

where the temperature T of the bath has been introduced according to the 
standard thermodynamic definition. 



dEb 



^ ■ (5.30) 



The term Sb{Ec) is a constant independent of the label a which can be absorbed 
into the normalization. We conclude that the distribution that codifies the 
relevant information about equilibrium is 



(5.31) 



5.4. THE CANONICAL FORMALISM 
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which we recognize as having the canonical form of eq.(4.66l. 

Our goal in this section was to identify the relevant variables. Here is the 
answer: the relevant information about thermal equilibrium can be summarized 
by the expected value of the energy (E) because someone who just knows (E) 
and is maximally ignorant about everything else is led to assign probabilities 
according to eq. (4.661 which coincides with (5.31 1. 

But our analysis has also disclosed an important limitation. Eq.(5.27l shows 
that in general the distribution for a system in equilibrium with a bath depends 
in a complicated way on the properties of the bath. The information in (E) 
is adequate only when the system and the bath interact weakly and the bath 
is so much larger than the system that its effects can be represented by a sin- 
gle parameter, the temperature T. Conversely, if these conditions are not met, 
then more information is needed. For example, the system might be sufficiently 
isolated that within the time scales of interest it can only reach thermal equi- 
librium with the few degrees of freedom in its very immediate vicinity. Then 
the surrounding bath need not be large and the information contained in the 
expected value (E) while still useful and relevant might just not be sufficient; 
more will be needed. 

Remark: The notion of relevance is relative. A particular piece of information 
might be relevant to one specific question and irrelevant to another. In the 
discussion above the system is in equilibrium, but we have not been sufficiently 
explicit about what specific questions one wants to address. It is implicit in this 
whole approach that one refers to the typical questions addressed in thermody- 
namics. 



5.4 The canonical formalism 

We consider a system (say, a fluid) in thermal equilibrium. The energy of the 
(conveniently discretized) microstate Za is Ea — Ea{V) where V is the volume of 
the system. We assume further that the expected value of the energy is known, 
{E)^E. ^ ^ 

Maximizing the (discretized) Gibbs entropy, 

S[p] = ~kJ2Pa log Pa where pa = f{za)Az , (5.32) 

a 

subject to constraints on normalization (1) — 1 and energy (E) — E yields, 



eq.(4.66) 



Pa = ^e-''^" (5.33) 



where the Lagrange multiplier f3 is determined from 

and Z{P,V)^j:^e-''^- . (5.34) 



d(3 



The maximized value of the Gibbs entropy is, eq.(4.69l, 



S{E, F) = fc log Z + k(3E . (5.35) 
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Differentiating with respect to E we obtain the analogue of eq.(4.77l, 



as 

dE 



dlog Z 8(3 dp _ 
" dp dE^^dE^"^ 



(5.36) 



where eq.(|5.34|) has been used to cancel the first two terms. In thermodynamics 

(5.37) 



temperature is defined by 



dE 



dcf 1 



therefore. 



(3- 



1 

kf 



(5.38) 



The connection between the formalism above and thermodynamics hinges on 
a suitable identification of work and heat. A small change in the internal energy 
5E can be induced by small changes in T and V, 



= EaPaSEa + Ea^aSpa ■ 



(5.39) 



Since Ea ~ Ea{V) the first term {5E) is an energy change that can be induced 
by small changes in volume, 



dE„ 



dE 



\ 



sv , 



this suggests that we can identify it with the mechanical work, 

(SE) =SW= -P6V , 
and therefore, the pressure is given by 



P 



dE 
dV 



This is the microscopic definition of pressure. 

The second term in eq.(5.39l must therefore represent heat, 

SQ = 6E - SW = 6 (E) - (SE) . 



(5.40) 



(5.41) 



(5.42) 



(5.43) 



The corresponding change in entropy is obtained from eq.(5.35l, 
SS 



= 51ogZ + S{(3E) 



1 



= — y 



-liEo. 



{EaSf3 + pdEa) + ESp + PSE 



^p{6E-{&E)) , 



therefore. 



(55 = 



5Q 
T 



(5.44) 
(5.45) 



This result is important. It proves that 
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The maximized Gihhs entropy, S{E,V), is identical to the thermodynamic 
entropy originally defined by Clausius. 



Substituting into eq.(5.43), yields the fundamental thermodynamic identity, 

6E ^ T5S - PSV . (5.46) 

Incidentally, it shows that the "natural" variables for energy are S and V, that 
is, E — E(S,V). Similarly, writing 



SS = -SE+-6V 
T T 



(5.47) 



confirms that S = S{E, V). 

The free energy F is defined by 



Z = e 



-PF 



Eq.(5.35l then leads to 



so that 



or F ^ -kT log Z{T,V) 



F ^ E-TS 



SF = -SST - PSV 



(5.48) 

(5.49) 
(5.50) 



which shows that F = F(T, V). 



Several useful thermodynamic relations can be easily obtained from eqs. (5.46 1 , 



(5.47), and (5.501. For example, the identities 



dF 
dV 



= -P, 



(5.51) 



can be read directly from eq.(5.50l. 



5.5 The Second Law of Thermodynamics 

We saw that in 1865 Clausius summarized the two laws of thermodynamics into 
"The energy of the universe is constant. The entropy of the universe tends to 
a maximum." We can be a bit more explicit about the Second Law: "In an 
adiabatic non-quasi-static process that starts and ends in equilibrium the total 
entropy increases; if the process is adiabatic and quasi-static process the total 
entropy remains constant." The Second Law was formulated in a somewhat 
stronger form by Gibbs (1878) "For irreversible processes not only does the 
entropy tend to increase, but it does increase to the maximum value allowed by 
the constraints imposed on the system." 

We are now ready to prove the Second Law. The proof below proposed by 
E. T. Jaynes in 1965 is mathematically very simple, but it is also conceptually 
subtle [Jaynes 65]. It may be useful to recall some of our previous results. The 
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entropy mentioned in the Second Law is the "thermodynamic" entropy St- It 
is defined only for equilibrium states by the Clausius relation, 



Sr{A)^J§ 

A J 



(5.52) 



where the integral is along a reversible path of intermediate equilibrium states. 
But as we saw in the previous section, in thermal equilibrium the maximized 
Gibbs entropy S'q^ - that is, the entropy computed from the canonical distri- 



bution - satisfies the same relation, eq.(5.45l 



T 



S'cf^iB) 



Sg^A) = / 



dQ 
T 



(5.53) 



If the arbitrary additive constant is adjusted so Sg*" matches St for one equi- 
librium state they will be equal for all equilibrium states. Therefore, if at any 
time t the system is in thermal equilibrium and its relevant macrovariables agree 
with expected values, say Xt, calculated using the canonical distribution then. 



ST{t) = 5-(t) 



(5.54) 



The system, which is assumed to be thermally insulated from its environ- 
ment, is allowed (or forced) to evolve according to a certain Hamiltonian. The 
evolution could, for example, be the free expansion of a gas into vacuum, or it 
could be given by the time-dependent Hamiltonian that describes some exter- 
nally prescribed influence, say, a moving piston or an imposed field. Eventually 
a new equilibrium is reached at some later time t'. Such a process is adiabatic; 
no heat was exchanged with the environment. Under these circumstances the 
initial canonical distribution /can(i), e.g. eq. (4.661 or (5.33), evolves according 
to Liouville's equation, eq.(|5.6|. 



fa 



.it) fit') 



(5.55) 



and, according to eq.(5.23l, the corresponding Gibbs entropy remains constant. 



5g^"(t) - Sait') 



(5.56) 



Since the Gibbs entropy remains constant it is sometimes argued that this 
contradicts the Second Law but note that the time-evolved Scit') is not the 
thermodynamic entropy because f{t') is not necessarily of the canonical form, 



From the new distribution fif) we can, however, compute the expected 
values Xf that apply to the state of equilibrium at t' . Of all distributions 
agreeing with the new values Xt' the canonical distribution /can(^') is that which 
has maximum Gibbs entropy, S'g'"(i'). Therefore 



Sait') < 5g^"(t') 



(5.57) 
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But So'^i^t') coincides with the thermodynamic entropy of the new equiHbrium 
state, 

S'g^"(t') = Srit') . (5.58) 



Collecting all these results, eqs.(5.54l-(5.58l, we conclude that the thermody- 
namic entropy has increased, 

Srit) < Srit') , (5.59) 

which is the Second Law. The equality applies when the time evolution is qua- 
sistatic so that throughout the process the distribution is always canonical; in 
particular, f{t') = /can(^')- The argument above can be generalized consider- 
ably by allowing heat exchanges or by introducing uncertainties into the actual 
Hamiltonian dynamics. 

To summarize, the chain of steps is 

Srit) = (<) = Soit') < (^') - Srit') . (5.60) 

(1) (2) (3) (4) 

Steps (1) and (4) hinge on identifying the maximized Gibbs entropy with the 
thermodynamic entropy - which works provided we have correctly identified 
the relevant macrovariables for the particular problem at hand. Step (2) follows 
from the constancy of the Gibbs entropy under Hamiltonian evolution - this is 
the least controversial step. Of course, if we did not have complete knowledge 
about the exact Hamiltonian H(t) acting on the system an inequality would 
have been introduced already at this point. The crucial inequality, however, is 
introduced in step (3) where information is discarded. The distribution f{t') 
contains information about the macrovariables Xf at time t' , and since the 
Hamiltonian is known, it also contains information about the values X-t the 
macrovariables had at the initial time t. In contrast, a description in terms of 
the distribution /can(i') contains information about the macrovariables Xt/ at 
time t' and nothing else. In a thermodynamic description all memory of the 
history of the system is lost. 

The Second Law refers to thermodynamic entropies only. These entropies 
measure the amount of information available to someone with only macroscopic 
means to observe and manipulate the system. The irreversibility implicit in the 
Second Law arises from this restriction to thermodynamic descriptions. 

It is important to emphasize what has just been proved: in an adiabatic 
process from one state of equilibrium to another the thermodynamic entropy 
increases. This is the Second Law. Many questions remain unanswered: We 
have assumed that the system tends towards and finally reaches an equilibrium; 
how do we know that this happens? What are the relaxation times, transport 
coefficients, etc.? There are all sorts of aspects of non-equilibrium irreversible 
processes that remain to be explained but this does not detract from what 
Jaynes' explanation did in fact accomplish, namely, it explained the Second 
Law, no more and, most emphatically, no less. 
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5.6 The thermodynamic Hmit 

If the Second Law "has only statistical certainty" (Maxwell, 1871) and any 
violation "seems to be reduced to improbability" (Gibbs, 1878) how can ther- 
modjmamic predictions attain so much certainty? Part of the answer hinges on 
restricting the kind of questions we are willing to ask to those concerning the few 
macroscopic variables over which we have some control. Most other questions 
are not "interesting" and thus they are never asked. For example, suppose we 
are given a gas in equilibrium within a cubic box, and the question is where will 
particle #23 be found. The answer is that we expect the particle to be at the 
center of the box but with a very large standard deviation - the particle can be 
anywhere in the box. The answer is not particularly impressive. On the other 
hand, if we ask for the energy of the gas at temperature T, or how it changes 
as the volume is changed by 6V, then the answers are truly impressive. 

Consider a system in thermal equilibrium in a macrostate described by a 
canonical distribution f{z) assigned on the basis of constraints on the values of 
certain macrovariables X. For simplicity we will assume X is a single variable, 
the energy, X = (E) = E. The microstates z can be divided into typical and 
atypical microstates. The typical microstates are all contained within a "high 
probability" region TZ^ to be defined below that has total probability 1 — e, 
where £ is a small positive number, and within which f{z) is greater than some 
lower bound. The "phase" volume of the typical region is 

Yo\{ne) = J^^dz = W, . (5.61) 

Our goal is to establish that the thermodynamic entropy and the volume of the 
region TZ^ are related through Boltzmann's equation, 

S'TWfclogWe. (5.62) 

The surprising feature is that the result is essentially independent of e. The fol- 
lowing theorems which are adaptations of the Asymptotic Equipartition Prop- 
erty (section 4.6) state this result in a mathematically precise way. 
Theorem: Let f{z) be the canonical distribution and kS = Sg = St the 
corresponding entropy, 

e-/3S(z) 

f{z) = = — and S = PE + logZ . (5.63) 

Then as iV ^ oo, 

1 S 
- — log f{z) — ^ in probability, (5.64) 

provided that the system is such that the energy fluctuations increase slower 
than N , that is, limjv^oo ^E/N = 0. (A denotes the standard deviation.) 

The theorem roughly means that 

The accessible microstates are essentially equally likely. 
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Microstatcs z for which (— log /(z))/iV differs substantially from S/N have ei- 
ther too low probability and are deemed "inaccessible," or they might individ- 
ually have a high probability but are too few to contribute significantly. 
Remark: The word 'essentially' is tricky because f{z) may differ from 
by a huge factor, but log/(2:) differs from —S by an unimportant amount that 
grows less rapidly than N. 

Remark: Note that the theorem applies only to those systems with interparti- 
cle interactions such that the energy fluctuations are sufficiently well behaved. 
Typically this requires that as N and V tend to infinity with N/V constant, the 
spatial correlations fall sufficiently fast that distant particles are uncorrelated. 
Under these circumstances energy and entropy are extensive quantities. 
Proof: Apply the Tchebyshev inequality (see section 2.8), 



to the variable 



P{\x-{x)\>e)<(^^^' , 



x= -j^ log f{z) . 



(5.65) 



(5.66) 



The mean is the entropy per particle, 



{x) = —{logf) 



To calculate the variance, 
{Axf 



{{\ogff)-{\ogfY 



use 



(log/)' 



{I3E + \og zy ^ 

0" (E^) + 2/3 (E) log Z + (log Zf 



so that 



Collecting these results gives 



Prob 



> £ 



< 



(5.67) 



(5.68) 



(5.69) 
(5.70) 

(5.71) 



For systems such that the relative energy fluctuations AE/E tend to as N~^^'^ 
when N oo, and the energy is an extensive quantity, E oc N, the limit on the 
right is zero, AE/N 0, therefore. 
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lim Prob 



-^log/(z)-| 



> e 



= , 



(5.72) 



which concludes the proof. 

The following theorem elaborates on these ideas further. To be precise let 
us define the typical region 7?,e as the set of microstates with probability /(z) 
such that 



or, using eq.(5.63 1, 



-0E+Ne 



(5.74) 



This last expression shows that typical microstates are those for which the en- 
ergy per particle E(z) /N lies within a narrow interval 2ekT about the expected 
value E/N. 

Remark: Even though some states z (namely those with energy E{z) < E) can 
individually be more probable than the typical states it turns out (see below) 
that they are too few and their volume is negligible compared to W^. 
Theorem of typical microstates: For N sufficiently large 

(1) Prob[7^e] > 1 - e 

(2) Vol(7^e) = We < e^+^^ 

(3) W, > (l-e)e'5-^^ 

(4) limAr^^(log W, - S)/N = 0. 
In words: 

The typical region has probability close to one; typical microstates are 
almost equally probable; the phase volume they occupy is about e^"^!^ , that 
is, St = k\ogW . 

The Gibbs entropy is a measure of the logarithm of the phase volume of typical 
states and for large N it does not much matter what we mean by typical (i.e., 
what we choose for e). Incidentally, note that it is the Gibbs entropy that 
satisfies the Boltzmann formula Sq = k log W. 

Proof: Eq.(5.72) states that for fixed e, for any given d there is an Ns such 
that for all N > Ng, we have 



Prob 



-^log/(^) 



S_ 
N 



< e 



> 1 



(5.75) 



Thus, the probability that the microstate z is e-typical tends to one, and there- 
fore so must Prob[7?,e]. Setting S ~ e yields part (1). This also shows that the 
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total probability of the set of states with E{z) < E is negligible - they must 
occupy a negligible volume. To prove (2) write 

1 > FToh[JZ,] = j'^dzf{z) 
> e-^-^'J^/z = e-^-^^W, . (5.76) 

Similarly, to prove (3) use (1), 

1 - e < Prob[7ee] = J-j^/zfiz) 

< e-^+^'^j^dz = e-^+^^W, , (5.77) 

Finally, from (2) and (3), 

(1 - e)e^-^' <We< e^+^' , (5.78) 

which is the same as 

S , , log(l-£) ^ logW, ^ S 

and proves (4). 

Remark: The theorems above can be generalized to situations involving several 

macrovariables X'' in addition to the energy. In this case, the expected value 
of log f{z) is 

(-log/) = 5 = Afc(X'=)+logZ, (5.80) 

and its variance is 

[Alogff = XkXm{{X>'X^)-{X''){X"^)) . (5.81) 



5.7 Interpretation of the Second Law: Repro- 
ducibility 

We saw that the Gibbs entropy is a measure of the logarithm of the phase 
volume of typical states. In the proof of the Second Law (section 4.11.1) we 
started with a system at time t in a state of thermal equilibrium defined by the 
macrovariables Xt- We saw (section 4.11.2) that within the typical region TZ{t) 
fluctuations of the Xt are negligible: all microstates are characterized by the 
same values of X. Furthermore, the typical region TZ{t) includes essentially all 
possible initial states compatible with the initial Xf. 

The volume W{t) = e'^^^*'/'' of the typical region can be interpreted in two 
ways. On one hand it is a measure of our ignorance as to the true microstate 
when all we know are the macrovariables Xf. On the other hand, the volume 
W{t) is also a measure of the extent that we can control the actual microstate 
of the system when the Xt are the only parameters we can manipulate. 

Having been prepared in equilibrium at time t the system is then subjected 
to an adiabatic process and it eventually attains a new equilibrium at time t'. 
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The Hamiltonian evolution deforms the initial region TZ(t) into a new region 
TZ{t') with exactly the same volume W{t) = W{t'); the macrovariables evolve 
from their initial values Xf to new values . 

Now suppose we adopt a thermodynamic description for the new equilibrium; 
the preparation history is forgotten, and all we know are the new values Xf>. 
The new typical region TZ'{t') has a volume W'{t') and it includes all microstates 
compatible with the information Xt'. 

After these preliminaries we come to the crux of the argument: With the 
limited experimental means at our disposal we can guarantee that the initial 
microstate will be somewhere within W{t) and therefore that in due course 
of time it will be within W{t'). In order for the process Xt X-^i to be 
experimentally reproducible it must be that all microstates in W{t') will also 
be within VK'(t') which means that W(t) = W{t') < W'{t'). Conversely, if 
it were true that W{t) > W'{t') we would sometimes observe that an initial 
microstate within W{t) would evolve into a final microstate lying outside W'{t') 
that is, sometimes we would observe X-^ X^'. Thus, when W{t) > W'{t') the 
experiment is not reproducible. 

A new element has been introduced into the discussion of the Second Law: 
reproducibility. [Jaynes 65] Thus, we can express the Second Law in the some- 
what tautological form: 

In a reproducible adiabatic process the thermodynamic entropy cannot de- 
crease. 

We can address this question from a different angle: How do we know that 
the chosen constraints X are the relevant macrovariables that provide an ade- 
quate thermodynamic description? In fact, what do we mean by an adequate 
description? Let us rephrase these questions differently: Could there exist addi- 
tional unknown physical constraints Y that significantly restrict the microstates 
compatible with the initial macrostate and which therefore provide an even bet- 
ter description? The answer is that such variables can, of course, exist but 
that including them in the description does not necessarily lead to an improve- 
ment. If the process Xt X^ is reproducible when no particular care has been 
taken to control the values of Y we can expect that to the extent that we are 
only interested in the X's the y's are irrelevant; keeping track of them will not 
yield a better description. Reproducibility is the criterion whereby we can decide 
whether a particular thermodynamic description is adequate or not. 

5.8 Remarks on irreversibility 

A considerable source of confusion on the question of reversibility is that the 
same word 'reversible' is used with several different meanings [Uffink 01]: 
(a) Mechanical or microscopic reversibility refers to the possibility of reversing 
the velocities of every particle. Such reversals would allow the system not just 
to retrace its steps from the final macrostate to the initial macrostate but it 
would also allow it to retrace its detailed microstate trajectory as well. 
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(b) Carnot or macroscopic reversibility refers to the possibility of retracing the 
history of 

macrostates of a system in the opposite direction. The required amount of 

control over the system can be achieved by forcing the system along a prescribed 
path of intermediate macroscopic equilibrium states that are infinitesimally close 
to each other. Such a reversible process is normally and appropriately called 
quasi-static. There is no implication that the trajectories of the individual 
particles will be retraced. 

(c) Thermodynamic reversibility refers to the possibility of starting from a final 
macrostate and completely recovering the initial macrostate without any other 
external changes. There is no need to retrace the intermediate macrostates in 
reverse order. In fact, rather than 'reversibility' it may be more descriptive to 
refer to 'recoverability' . Typically a state is irrecoverable when there is friction, 
decay, or corruption of some kind. 

Notice that when one talks about the "irreversibility" of the Second Law and 
about the "reversibility" of mechanics there is no inconsistency or contradiction: 
the word 'reversibility' is being used with two entirely different meanings. 

Classical thermodynamics assumes that isolated systems approach and even- 
tually attain a state of equilibrium. The state of equilibrium is. by definition, 
a state that, once attained, will not spontaneously change in the future. On 
the other hand, it is understood that changes might have happened in the past. 
Classical thermodynamics introduces a time asymmetry: it treats the past and 
the future differently. 

The situation with statistical mechanics is, however, somewhat different. 
Once equilibrium has been attained fluctuations are possible. In fact, if we wait 
long enough we can expect that large fluctuations can be expected to happen in 
the future, just as they might have happened in the past. The situation is quite 
symmetric. The interesting asymmetry arises when we realize that for a large 
fluctuation to happen spontaneously in the future might require an extremely 
long time while we just happen to know that a similarly large "fluctuation" was 
observed in the very recent past. This might seem strange because the formalism 
of statistical mechanics does not introduce any time asymmetry. The solution 
to the puzzle is that the large "fluctuation" in the recent past most likely did 
not happen spontaneously but was quite deliberately brought about by human 
(or otherwise) intervention. The system was prepared in some unusual state by 
applying appropriate constraints which were subsequently removed - we do this 
all the time. 

5.9 Entropies, descriptions and the Gibbs para- 
dox 

Under the generic title of "Gibbs Paradox" one usually considers a number of 
related questions in both phenomenological thermodynamics and in statistical 
mechanics: (1) The entropy change when two distinct gases are mixed happens 
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to be independent of the nature of the gases. Is this in conflict with the idea 
that in the limit as the two gases become identical the entropy change should 
vanish? (2) Should the thermodynamic entropy of Clausius be an extensive 
quantity or not? (3) Should two microstates that differ only in the exchange of 
identical particles be counted as two or just one microstate? 

The conventional wisdom asserts that the resolution of the paradox rests 
on quantum mechanics but this analysis is unsatisfactory; at best it is incom- 
plete. While it is true that the exchange of identical quantum particles does 
not lead to a new microstate this approach ignores the case of classical, and 
even non-identical particles. For example, nanoparticles in a colloidal suspen- 
sion or macromolecules in solution are both classical and non-identical. Several 
authors (e.g., [Grad 61, Jaynes 92]) have recognized that quantum theory has 
no bearing on the matter; indeed, as remarked in section 3.5, this was already 
clear to Gibbs. 

Our purpose here is to discuss the Gibbs paradox from the point of view of 
information theory. The discussion follows [Tseng Caticha 01]. Our conclusion 
will be that the paradox is resolved once it is realized that there is no such thing 
as the entropy of a system, that there are many entropies. The choice of entropy 
is a choice between a description that treats particles as being distinguishable 
and a description that treats them as indistinguishable; which of these alterna- 
tives is more convenient depends on the resolution of the particular experiment 
being performed. 

The "grouping" property of entropy, eq.(4.3l, 

plays an important role in our discussion. It establishes a relation between two 
different descriptions and refers to three different entropies. One can describe 
the system with high resolution as being in a microstate i (with probability 
Pi), or alternatively, with lower resolution as being in one of the groups g (with 
probability Pg). Since the description in terms of the groups g is less detailed 
we might refer to them as 'mesostates'. A thermodynamic description, on the 
other hand, corresponds to an even lower resolution that merely specifies the 
equilibrium macrostate. For simplicity, we will define the macrostate with a 
single variable, the energy. Including additional variables is easy and does not 
modify the gist of the argument. 

The standard connection between the thermodynamic description in terms of 
macrostates and the description in terms of microstates is established in section 
4.10.4. If the energy of microstate a is Ea, to the macrostate of energy E — (E) 
we associate that canonical distribution (5.331 

Pa = , (5.82) 

where the partition function Zh and the Lagrange multiplier /3 are determined 
from eqs. (5.341, 

ZH = j:e-'^^ and - -E . (5.83) 
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The corresponding entropy, eq.(5.35l is (setting fc = 1) 



(5.84) 



measures the amount of information required to specify the microstate when all 
we know is the value E . 

Identical particles 

Before we compute and interpret the probability distribution over mesostates 
and its corresponding entropy we must be more specific about which mesostates 
we are talking about. Consider a system of N classical particles that are exactly 
identical. The interesting question is whether these identical particles are also 
"distinguishable." By this we mean the following: we look at two particles now 
and we label them. We look at the particles later. Somebody might have 
switched them. Can we tell which particle is which? The answer is: it depends. 
Whether we can distinguish identical particles or not depends on whether we 
were able and willing to follow their trajectories. 

A slightly different version of the same question concerns an A^-particle sys- 
tem in a certain state. Some particles are permuted. Does this give us a different 
state? As discussed earlier the answer to this question requires a careful speci- 
fication of what we mean by a state. 

Since by a microstate we mean a point in the iV-particle phase space, then 
a permutation does indeed lead to a new microstate. On the other hand, our 
concern with permutations suggests that it is useful to introduce the notion of 
a mesostate defined as the group of those iV! microstates that are obtained as 
permutations of each other. With this definition it is clear that a permutation 
of the identical particles does not lead to a new mesostate. 

Now we can return to discussing the connection between the thermodynamic 
macrostate description and the description in terms of mesostates using, as 
before, the Method of Maximum Entropy. Since the particles are (sufliciently) 
identical, all those A^! microstates i within the same mesostate g have the same 
energy, which we will denote by Eg (i.e., Ei = Eg for all i £ g). To the 
macrostate of energy E = (E) we associate the canonical distribution, 



e 



Zl 



(5.85) 



where 





= -E. 



(5.86) 



The corresponding entropy, eq.(5.35l is (setting fc = 1) 



SL = PE + logZL, 



(5.87) 



measures the amount of information required to specify the mesostate when all 
we know is E. 
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Two different entropies Sh and Sl have been assigned to the same macrostate 
E; they measure the different amounts of additional information required to 
specify the state of the system to a high resolution (the microstate) or to a low 
resolution (the mesostate). 

The relation between and Zj^ is obtained from 

= Ee"^""' = ^iEe^'"''' - or = §f ■ (5.88) 

i g -'^ • 

The relation between Sh and Sl is obtained from the "grouping" property, 



eq.(4.3l, with S = Sh and Sq = Sl, and pi\g = 1/N\. The result is 

Sl^ Sn-logm. (5.89) 

Incidentally, note that 

Sh = -EaPalogPa = .Pg^og Pg / Nl . (5.90) 



Equations (5.88 1 and (5.89 1 both exhibit the Gibbs N\ "corrections." Our analy- 
sis shows (1) that the justification of the N\ factor is not to be found in quantum 
mechanics, and (2) that the TV! does not correct anything. The N\ is not a fudge 
factor that fixes a wrong (possibly nonextensive) entropy Sh into a correct (pos- 
sibly extensive) entropy 5*^. Both entropies Sh and Sl are correct. They differ 
because they measure different things: one measures the information to specify 
the microstate, the other measures the information to specify the mesostate. 

An important goal of statistical mechanics is to provide a justification, an 
explanation of thermodynamics. Thus, we still need to ask which of the two 
statistical entropies, Sh or 5*^, should be identified with the thermodynamic 



entropy of Clausius St- Inspection of eqs. (5.881 and (5.891 shows that, as long 
as one is not concerned with experiments that involve changes in the number 
of particles, the same thermodynamics will follow whether we set Sh — St or 
Sl = St- 

But, of course, experiments involving changes in N are very important (for 
example, in the equilibrium between different phases, or in chemical reactions). 
Since in the usual thermodynamic experiments we only care that some number 
of particles has been exchanged, and we do not care which were the actual par- 
ticles exchanged, we expect that the correct identification is Sl = St- Indeed, 
the quantity that regulates the equilibrium under exchanges of particles is the 
chemical potential defined by 

f^--kT('^] (5.91) 



V dN 



E,V,... 



The two identifications Sh = St or Sl = St, lead to two different chemical 
potentials, related by 

^lL=m-NkT. (5.92) 

It is easy to verify that, under the usual circumstances where surface effects 
can be neglected relative to the bulk, ^l has the correct functional dependence 
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on N: it is intensive and can be identified with the thermodynamic /i. On the 
other hand, ijh is not an intensive quantity and cannot therefore be identified 
with /i. 

Non-identical particles 

We saw that classical identical particles can be treated, depending on the res- 
olution of the experiment, as being distinguishable or indistinguishable. Here 
we go further and point out that even non-identical particles can be treated as 
indistinguishable. Our goal is to state explicitly in precisely what sense it is up 
to the observer to decide whether particles are distinguishable or not. 

We defined a mesostatc as a subset of A^! microstatcs that are obtained as 
permutations of each other. With this definition it is clear that a permutation 
of particles does not lead to a new mesostate even if the exchanged particles 
are not identical. This is an important extension because, unlike quantum 
particles, classical particles cannot be expected to be exactly identical down to 
every minute detail. In fact in many cases the particles can be grossly different - 
examples might be colloidal suspensions or solutions of organic macromoleculcs. 
A high resolution device, for example an electron microscope, would reveal that 
no two colloidal particles or two macromolecules are exactly alike. And yet, 
for the purpose of modelling most of our macroscopic observations it is not 
necessary to take account of the myriad ways in which two particles can differ. 

Consider a system of N particles. We can perform rather crude macroscopic 
experiments the results of which can be summarized with a simple phenomeno- 
logical thermodynamics where A'^ is one of the relevant variables that define the 
macrostate. Our goal is to construct a statistical foundation that will explain 
this macroscopic model, reduce it, so to speak, to "first principles." The par- 
ticles might ultimately be non-identical, but the crude phenomenology is not 
sensitive to their differences and can be explained by postulating mesostates g 
and microstatcs * with energies Ei w Eg, for all i e g, as if the particles were 
identical. As in the previous section this statistical model gives 

Zl = ^ with ZH = ^e-'5^S (5.93) 

i 

and the connection to the thermodynamics is established by postulating 

St = Sl = Sh- log m . (5.94) 

Next we consider what happens when more sophisticated experiments are 
performed. The examples traditionally offered in discussions of this sort refer to 
the new experiments that could be made possible by the discovery of membranes 
that are permeable to some of the N particles but not to the others. Other, 
perhaps historically more realistic examples, are afforded by the availability 
of new experimental data, for example, more precise measurements of a heat 
capacity as a func;tion of temperature, or perhaps measurements in a range of 
temperatures that had previously been inaccessible. 
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Suppose the new phenomenology can be modelled by postulating the exis- 
tence of two kinds of particles. (Experiments that are even more sophisticated 
might allow us to detect three or more kinds, perhaps even a continuum of 
different particles.) What we previously thought were N identical particles we 
will now think as being N^, particles of type a and Nf, particles of type b. The 
new description is in terms of macrostates defined by Na and Nf, as the relevant 
variables. 

To construct a statistical explanation of the new phenomenology from 'first 
principles' we need to revise our notion of mesostate. Each new mesostate will 
be a group of microstates which will include all those microstates obtained by 
permuting the a particles among themselves, and by permuting the b particles 
among themselves, but will not include those microstates obtained by permuting 
a particles with b particles. The new mesostates, which we will label g and to 
which we will assign energy Eg, will be composed of NJNbl microstates i, each 
with a well defined energy E^ = Eg, for all i G g. The new statistical model 
gives 



and the connection to the new phenomenology is established by postulating 



In discussions of this topic it is not unusual to find comments to the effect 

that in the limit as particles a and b become identical one expects that the 
entropy of the system with two kinds of particles tends to the entropy of a 
system with just one kind of particle. The fact that this expectation is not met 
is one manifestation of the Gibbs paradox. 

From the information theory point of view the paradox does not arise because 
there is no such thing as the entropy of the system, there are several entropies. 
It is true that as a ^ & we will have Zh Zh, and accordingly Sh — > Sh, 
but there is no reason to expect a similar relation between and because 
these two entropies refer to mesostates g and g that remain different even as 
a and b became identical. In this limit the mesostates g, which are useful for 
descriptions that treat particles a and b as indistinguishable among themselves 
but distinguishable from each other, lose their usefulness. 

Conclusion 

The Gibbs paradox in its various forms arises from the widespread misconception 
that entropy is a real physical quantity and that one is justified in talking about 
the entropy of the system. The thermodynamic entropy is not a property of the 
system. Entropy is a property of our description of the system, it is a property of 
the macrostate. More explicitly, it is a function of the macroscopic variables used 
to define the macrostate. To different macrostates reflecting different choices of 
variables there correspond different entropies for the very same system. 



Zl 




with 




(5.95) 



St = Sl = Sh- log NJNil. 



(5.96) 
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But this is not the complete story: entropy is not just a function of the 
macrostate. Entropies reflect a relation between two descriptions of the same 
system: in addition to the macrostate, we must also specify the set of mi- 
crostatcs, or the set of mcsostatcs, as the case might be. Then, having specified 
the macrostate, an entropy can be interpreted as the amount of additional in- 
formation required to specify the microstate or mesostate. We have found the 
'grouping' property very valuable precisely because it emphasizes the depen- 
dence of entropy on the choice of micro or mesostates. 



Chapter 6 

Entropy III: Updating 
Probabilities 

The general problem of inductive inference is to update from a prior probability 
distribution to a posterior distribution when new information becomes available. 
The challenge is to develop updating methods that are both systematic and 
objective. In Chapter 2 we saw that Bayes' rule is the natural way to update 
when the information is in the form of data. We also saw that Bayes' rule could 
not be derived just from the requirements of consistency implicit in the sum 
and product rules of probability theory. An additional Principle of Minimal 
Updating (PMU) was necessary: Prior information is valuable and should not 
be discarded; beliefs should be revised only to the extent required by the data. A 
few interesting questions were just barely hinted at: How do we update when 
the information is not in the form of data? If the information is not data, what 
else could it possibly be? Indeed what, after all, is information? 

Then in Chapter 4 we saw that the method of maximum entropy, MaxEnt, 
allowed one to deal with information in the form of constraints on the allowed 
probability distributions. So here we have a partial answer to one of our ques- 
tions: in addition to data information can take the form of constraints. However, 
MaxEnt is not a method for updating; it is a method for assigning probabilities 
on the basis of the constraint information, but it does not allow us to take into 
account the information contained in prior distributions. 

Thus, Bayes' rule allows for the information contained in arbitrary priors 
and in data, but not in arbitrary constraints]^ while on the other hand, Max- 
Ent can handle arbitrary constraints but not arbitrary priors. In this chapter 
we bring those two methods together: by generalizing the PMU we show how 
the MaxEnt method can be extended beyond its original scope, as a rule to 
assign probabilities, to a full-fledged method for inductive inference, that is, 
a method for updating from arbitrary priors given information in the form of 

^ Bayes' rule can handle constraints when they are expressed in the form of data that can 
be plugged into a likelihood function. Not all constraints are of this kind. 
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arbitrary constraints. It should not be too surprising that the extended Max- 
imum Entropy method, which we will henceforth abbreviate as ME, includes 
both MaxEnt and Bayes' rule as special cases. 

Historically the ME method is a direct descendant of MaxEnt. As we saw 
in chapter 4 within the MaxEnt method entropy is interpreted through the 
Shannon axioms as a measure of the amount of uncertainty or of the amount 
of information that is missing in a probability distribution. We discussed some 
limitations of this approach. The Shannon axioms refer to probabilities of dis- 
crete variables; for continuous variables the entropy is not defined. But a more 
serious objection was raised: even if we grant that the Shannon axioms do lead 
to a reasonable expression for the entropy, to what extent do we believe the 
axioms themselves? Shannon's third axiom, the grouping property, is indeed 
sort of reasonable, but is it necessary? Is entropy the only consistent measure 
of uncertainty or of information? What is wrong with, say, the standard de- 
viation? Indeed, there exist examples in which the Shannon entropy does not 
seem to reflect one's intuitive notion of information [Uffink 95]. Other entropies, 
justified by a different choice of axioms, can be introduced (prominent examples 
are [Renyi 61, Tsallis 88]). 

From our point of view the real limitation is that neither Shannon nor Jaynes 
were concerned with updating probabilities. Shannon was analyzing the ca- 
pacity of communication channels and characterizing the potential diversity of 
messages generated by information sources (section 4.6). His entropy makes no 
reference to prior distributions. On the other hand, as we already mentioned, 
Jaynes conceived MaxEnt as a method to assign probabilities on the basis of 
constraint information and a fixed underlying measure, not an arbitrary prior. 
He never meant to update from one probability distribution to another. 

Considerations such as these motivated several attempts to develop ME di- 
rectly as a method for updating probabilities without invoking questionable 
measures of uncertainty. Prominent among them are [Shore and Johnson 80, 
Skilling 88-90, Csiszar 91]. The important contribution by Shore and Johnson 
was the realization that one could axiomatize the updating method itself rather 
than the information measure. Their axioms are justified on the basis of a fun- 
damental principle of consistency if a problem can be solved in more than one 
way the results should agree but the axioms themselves and other assump- 
tions they make have raised some objections [Karbelkar 86, Uffink 95]). Despite 
such criticism Shore and Johnson's pioneering papers have had an enormous 
influence; they identifled the correct goal to be achieved. 

Another approach to entropy was proposed by Skilling. His axioms are 
clearly inspired by those of Shore and Johnson but his approach is different in 
several important aspects, in particular Skilling did not explore the possibility 
of using his induction method for the purpose for inductive inference, that is, 
for updating from prior to posterior probabilities. 

The primary goal of this chapter is to apply Skilling's method of eliminative 
induction to Shore and Johnson's problem of updating probabilities and, in 
the process, to overcome the objections that can be raised against either. The 
presentation below follows [Caticha 03, Caticha Giffin 06, Caticha 07]. 
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As we argued earlier when developing the theory of degrees of belief, our 
general approach differs from the way in which many physical theories have been 
developed in the past. The more traditional approach consists of first setting 
up the mathematical formalism and then seeking an acceptable interpretation. 
The drawback of this procedure is that questions can always be raised about the 
uniqueness of the proposed interpretation, and about the criteria that makes it 
acceptable or not. 

In contrast, here we proceed in the opposite order: we first decide what we 
are talking about, what goal we want to achieve, and only then we design a 
suitable mathematical formalism. The advantage is that the issue of meaning 
and interpretation is resolved from the start. The preeminent example of this 
approach is Cox's algebra of probable inference (discussed in chapter 2) which 
clarified the meaning and use of the notion of probability: after Cox it was 
no longer possible to raise doubts about the legitimacy of the degree of belief 
interpretation. A second example is special relativity: the actual physical sig- 
nificance of the X and t appearing in the mathematical formalism of Lorentz and 
Poincare was a matter of controversy until Einstein settled the issue by deriving 
the formalism,that is, the Lorentz transformations, from more basic principles. 
Yet a third example is the derivation of the mathematical formalism of quantum 
theory. [Caticha 98] In this chapter we explore a fourth example: the concept of 
relative entropy is introduced as a tool for reasoning which reduces to the usual 
entropy in the special case of uniform priors. There is no need for an interpre- 
tation in terms of heat, multiplicity of states, disorder, uncertainty, or even in 
terms of an amount of information. In this approach we find an explanation 
for why the search for the meaning of entropy has turned out to be so elusive: 
Entropy needs no interpretation. We do not need to know what 'entropy' means; 
we only need to know how to use it. 

Since the PMU is the driving force behind both Bayesian and ME updating 
it is worthwhile to investigate the precise relation between the two. We show 
that Bayes' rule can be derived as a special case of the ME methodj^ The virtue 
of our derivation, which hinges on translating information in the form of data 
into a constraint that can be processed using ME, is that it is particularly clear. 
It throws light on Bayes' rule and demonstrates its complete compatibility with 
ME updating. A slight generalization of the same ideas shows that Jeffrey's 
updating rule (section 2.10.2) is also a special case of the ME method. Thus, 
within the ME framework maximum entropy and Bayesian methods are unified 
into a single consistent theory of inference. 

There is a second function that the ME method must perform in order to 
fully qualify as a method of inductive inference: once we have decided that the 
distribution of maximum entropy is to be preferred over all others the following 
question arises immediately: the maximum of the entropy function is never 
infinitely shaxp, are we really confident that distributions with entropy very 
close to the maximum are totally ruled out? We must find a quantitative way 

■^This result was first obtained by WiUiams (see [Williams 80, Diaconis 82]) long before the 
logical status of the ME method - and therefore the full extent of its implications — had been 
sufficiently clarified. 
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to assess the extent to which distributions with lower entropy are ruled out. 
This matter is addressed following the treatment in [Caticha 00]. 

6.1 What is information? 

It is not unusual to hear that systems "carry" or "contain" information and that 
"information is physical" . This mode of expression can perhaps be traced to the 
origins of information theory in Shannon's theory of communication. We say 
that we have received information when among the vast variety of messages that 
could conceivably have been generated by a distant source, we discover which 
particular message was actually sent. It is thus that the message "carries" 
information. The analogy with physics is straightforward: the set of all possible 
states of a physical system can be likened to the set of all possible messages, 
and the actual state of the system corresponds to the message that was actually 
sent. Thus, the system "conveys" a message: the system "carries" information 
about its own state. Sometimes the message might be difficult to read, but it is 
there nonetheless. 

This language - information is physical - useful as it has turned out to be, 
does not exhaust the meaning of the word 'information'. The goal of informa- 
tion theory, or better, communication theory, is to characterize the sources of 
information, to measure the capacity of the communication channels, and to 
learn how to control the degrading effects of noise. It is somewhat ironic but 
nevertheless true that this "information" theory is unconcerned with the central 
Bayesian issue of how the message affects the beliefs of a rational agent. 

A fully Bayesian information theory demands an explicit account of the re- 
lation between information and beliefs. 

The notion that the theory for reasoning with incomplete information is the 
theory of degrees of rational belief led us to tackle two different problems]^ The 
first was to understand the conditions required to achieve consistency within a 
web of interconnected beliefs. This problem was completely solved: degrees of 
belief are consistent when they obey the rules of probability theory, which led 
us to conclude that rational degrees of belief are probabilities. 

The second problem is that of updating probabilities when new information 
becomes available. The desire and need to update our beliefs is driven by the 
conviction that not all probability assignments are equally good. This bears 
on the issue of whether probabilities are subjective, objective, or somewhere in 
between. We argued earlier that what makes one probability assignment better 
than another is that it better reflects some "objective" feature of the world, 
that is, it provides a better guide to the "truth" - whatever this might mean. 
Therefore objectivity is a desirable goal. It is their (partial) objectivity that 
makes probabilities useful. Indeed, what we seek are updating mechanisms that 



^We mentioned earlier, and emphasize again here, that the quahfier 'rational' is crucial: 
we are interested in the reasoning of an idealized rational agent and not of real imperfect 
humans. 
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allow us to process information and incorporate its objective features into our 
beliefs. 

Bayes' rule behaves precisely in this way. We saw in section 2.10 that as 

more and more data are taken into account the original (possibly subjective) 
prior becomes less and less relevant, and all rational agents become more and 
more convinced of the same truth. This is crucial: were it not this way Bayesian 
reasoning would not be deemed acceptable. 

We are now ready to answer the question 'What, after all, is information?' 
The result of being confronted with new information should be a restriction 
on our options as to what wc arc honestly and rationally allowed to believe. 
This, I propose, is the defining characteristic of information. By information, 
in its most general form, I mean a set of constraints on the family of acceptable 
posterior distributions. Thus, 

Information is whatever constrains rational beliefs. 

We can phrase this idea somewhat differently. Since our objective is to update 
from a prior distribution to a posterior when new information becomes available 

wc can state that 

Information is what forces a change of beliefs. 

An important aspect of this notion is that for a rational agent the updating is 
not optional: it is a moral imperative. 

Our definition captures an idea of information that is directly related to 
changing our minds: information is the driving force behind the process of 
learning. Note also that although there is no need to talk about amounts of 
information, whether measured in units of bits or otherwise, our notion of in- 
formation allows precise quantitative calculations. Indeed, constraints on the 
acceptable posteriors are precisely the kind of information the method of max- 
imum entropy (see below) is designed to handle. 

The constraints that convey, or rather, that arc information can take a wide 
variety of forms. For example, they can represent data (see section 6.5 below), 
or they can be in the form of expected values (as in statistical mechanics, see 
chapter 5). Although one cannot directly measure expected values or prob- 
abilities one can still use them to convey information. This is what we do, 
for example, when we specify a prior or the likelihood function - this is not 
something that one can measure but by constraining our beliefs they certainly 
are valuable information. Constraints can also be specified through geometrical 
relations (see section 6.7 and also [Caticha 01, Caticha Cafaro 07]). 

It may be worthwhile to point out an analogy with dynamics - the study of 
change. In Newtonian dynamics the state of motion of a system is described in 
terms of momentum the "quantity" of motion - while the change from one 
state to another is explained in terms of an applied force. Similarly, in Bayesian 
inference a state of belief is described in terms of probabihties - a "quantity" 
of belief - and the change from one state to another is due to information. Just 
as a force is defined as that which induces a change from one state of motion to 
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another, so information is that which induces a change from one state of belief 
to another. 

What about prejudices and superstitions? What about divine revelations? 

Do they constitute information? Perhaps they lie outside our chosen subject of 
ideally rational beliefs, but to the extent that their effects are indistinguishable 
from those of other sorts of information, namely, they affect beliefs, they qualify 
as information too. Whether the sources of such information are reliable or not 
is quite another matter. False information is information too and even ideally 
rational agents are affected by false information. 

What about limitations in our computational power? They influence our 
inferences. Should they be considered information? No. Limited computational 
resources may affect the numerical approximation to the value of, say, an inte- 
gral, but they do not affect the actual value of the integral. Similarly, limited 
computational resources may affect the approximate imperfect reasoning of real 
humans and real computers but they do not affect the reasoning of those ideal 
rational agents that are the subject of our present concerns. 

6.2 Entropy as a tool for updating probabilities 

Consider a variable x the value of which is uncertain. The variable can be 
discrete or continuous, in one or in several dimensions. For example, x could 
represent the possible microstates of a physical system, a point in phase space, or 
an appropriate set of quantum numbers. The uncertainty about x is described by 
a probability distribution q{x). Our goal is to update from the prior distribution 
q{x) to a posterior distribution p{x) when new information - by which we mean 
a set of constraints - becomes available. The information can be given in terms 
of expected values but this is not necessary. The question is: of all those 
distributions within the family defined by the constraints, what distribution 
p{x) should we select? 

To select the posterior one could proceed by attempting to place all candi- 
date distributions in increasing order of preference. [SkilUng 88] Irrespective of 
what it is that makes one distribution preferable over another it is clear that 
any ranking according to preference must be transitive: if distribution pi is pre- 
ferred over distribution p2, and P2 is preferred over ^33, then pi is preferred over 
P3. Such transitive rankings are implemented by assigning to each p{x) a real 
number S\p] in such a way that if pi is preferred over p2, then S\pi] > S[p2]. 
The selected distribution (one or possibly many, for there may be several equally 
preferred distributions) will be that wliic;h maximizes the functional S[p\ which 
we will call the entropy of p. We are thus led to a method of Maximum Entropy 
(ME) that is a variational method involving entropies which are real numbers. 
These are features imposed by design; they are dictated by the function that 
the ME method is supposed to perform. 

Next, to define the ranking scheme, we must decide on the functional form of 
S\p]. First, the purpose of the method is to update from priors to posteriors. The 
ranking scheme must depend on the particular prior q and therefore the entropy 
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S must be a functional of both p and q. The entropy S[p, q] describes a ranking 
of the distributions p relative to the given prior q. S\p, q] is the entropy of p 
relative to q, and accordingly S\p, q] is commonly called relative entropy. This 
is appropriate and sometimes we will follow this practice. However, as discussed 
in section 4.5, even the 'regular' Shannon entropy is relative, it is the entropy of 
p relative to an underlying uniform distribution. Since all entropies are relative 
to some prior, the qualifier 'relative' and is redundant can be dropped. This 
is somewhat analogous to the situation with energy: all energies are relative 
to some origin or to some reference frame but we do not feel compelled to 
constantly refer to the 'relative energy'. It is just taken for granted. 

Second, since we deal with incomplete information the method, by its very 
nature, cannot be deductive: the method must be inductive. The best we can 
do is generalize from those few special cases where we know what the preferred 
distribution should be to the much larger number of cases where we do not. In 
order to achieve its purpose, we must assume that S[p, q] is of universal applica- 
bility. There is no justification for this universality beyond the usual pragmatic 
justification of induction: in order to avoid the paralysis of not generalizing at 
all we must risk making wrong generalizations. An induction method must be 
allowed to induce. 

We will apply the Principle of Eliminative Induction introduced in 
chapter 1: 

If a general theory exists it must apply to special cases. 

If special examples are known then all candidate theories that fail to re- 
produce the known examples are discarded. 

If a sufficient number of special examples are known then the general 
theory might be completely determined. 

The best we can do is use those special cases where we know what the preferred 
distribution should be to eliminate those entropy functionals g] that fail 
to provide the right update. The known special cases will be called (perhaps 
inappropriately) the axioms of the theory. They play a crucial role: they define 
what makes one distribution preferable over another. 

The three axioms below are chosen to reflect the moral conviction that in- 
formation collected in the past and codified into the prior distribution is very 
valuable and should not be frivolously discarded. This attitude is radically con- 
servative: the only aspects of one's beliefs that should be updated are those for 
which new evidence has been supplied. This is important and it is worthwhile to 
consider it from a different angle. Degrees of belief, probabilities, are said to be 
subjective: two different individuals might not share the same beliefs and could 
conceivably assign probabilities differently. But subjectivity does not mean ar- 
bitrariness. It is not a blank check allowing the rational agent to change its 
mind for no good reason. Valuable prior information should not be discarded 
until new information renders it obsolete. 
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Furthermore, since the axioms do not tell us what and how to update, they 
merely tell us what not to update, they have the added bonus of maximizing 
objectivity - there are many ways to change something but only one way to 
keep it the same. Thus, we adopt the 

Principle of Minimal Updating (PMU): Beliefs should be updated only 
to the extent required by the new information. 

The three axioms, the motivation behind them, and their consequences for the 
functional form of the entropy functional are given below. As will become im- 
mediately apparent the axioms do not refer to merely three cases; any induction 
from such a weak foundation would hardly be reliable. The reason the axioms 
are convincing and so constraining is that they refer to three infinitely large 
classes of known special cases. Detailed proofe are deferred to the next section. 

Axiom 1: Locality. Local information has local effects. 
Suppose the information to be processed does not refer to a particular subdo- 
main T> of the space X of xs. In the absence of any new information about 
D the PMU demands we do not change our minds about V. Thus, we design 
the inference method so that q{x\'D), the prior probability of x conditional on 
X G V, is not updated. The selected conditional posterior is P{x\'D) = q{x\'D). 
We emphasize: the point is not that we make the unwarranted assumption that 
keeping q{x\T>) is guaranteed to lead to correct inferences. It need not. Induc- 
tion is risky. The point is, rather, that in the absence of any evidence to the 
contrary there is no reason to change our minds and the prior information takes 
priority. 

The consequence of axiom 1 is that non-overlapping domains of x contribute 
additively to the entropy. 



where F is some unknown function - not a functional, just a regular function 
of three arguments. 

Axiom 2: Coordinate invariance. The system of coordinates carries no 
information. 

The points x can be labeled using any of a variety of coordinate systems. In 
certain situations we might have explicit reasons to believe that a particular 
choice of coordinates should be preferred over others. This information might 
have been given to us in a variety of ways, but unless the evidence was in fact 
given we should not assume it: the ranking of probability distributions should 
not depend on the coordinates used. 

To grasp the meaning of this axiom it may be useful to recall some facts 
about coordinate transformations. Consider a change from old coordinates x to 
new coordinates x' such that x = r(a;'). The new volume element dx' includes 
the corresponding Jacobian, 




(6.1) 



dx = j{x')dx' where jix') = 



dx 



(6.2) 



dx' 
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Let m{x) be any density; the transformed density m'(x') is such that m.{x)dx = 
m!{x')dx' . This is true, in particular, for probabiUty densities such as p{x) and 
q{x), therefore 

, , m'(x') , , p'fa;') , , , (i{x'^ 
^{x') l{x ) 7{x ) 

The coordinate transformation gives 



S\p,q]= J dxF{p{x),q{x),x) 

= /.M<ix'F(^,^,r(.')). (6.4) 

which is a mere change of variables. The identity above is valid always, for all T 
and for all F; it imposes absolutely no constraints on S[p, q]. The real constraint 
arises from realizing that we could have started in the x' coordinate frame, in 
which case we would have have ranked the distributions using the entropy 

S\p',q'] = J dx'F{p'{x'),q'{x'),x') , (6.5) 

but this should have no effect on our conclusions. This is the nontrivial content 
of axiom 2. It is not that we can change variables, we can always do that; but 
rather that the two rankings, the one according to S[p, q] and the other according 
to S[p',q'] must coincide. This requirement is satisfied if, for example, S\p,q] 
and S[p', q'] turn out to be numerically equal, but this is not necessary. 

The consequence of axiom 2 is that S\p, q] can be written in terms of coor- 
dinate invariants such as dxm{x) and p{x)/m{x), and q{x)/m{x): 

S[p,,l = /<,xmW*(ig.,^) . 

Thus the unknown function F which had three arguments has been replaced by 
a still unknown function $ with two arguments plus an unknown density m{x). 

Next we determine the density m(.x) by invoking the locality axiom 1 once 
again. A situation in which no new information is available is dealt by allowing 
the domain T> to cover the whole space X. The requirement that in the absence 
of any new information the prior conditional probabilities q{x\'D) = q{x\X) = 
pqx) should not be updated, can be expressed as 

Axiom 1 (special case): When there is no new information there is no 
reason to change one 's mind. 

When there are no constraints the selected posterior distribution should coincide 
with the prior distribution, that is, P{x) = q{x). The consequence of this second 
use of locality is that the arbitrariness in the density m(x) is removed: up to 
normalization m{x) must be the prior distribution q{x), and therefore at this 
point we have succeeded in restricting the entropy to functionals of the form 

S\p,q] = J dxq{x)<^(^^'j (6.7) 
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Axiom 3: Consistency for independent subsystems. When a system is 
composed of subsystems that are known to be independent it should not matter 
whether the inference procedure treats them separately or jointly. 

This axiom is perhaps subtler than it appears at first sight. Two points must 
be made clear. The first point concerns how the information about independence 
is to be handled as a constraint. Consider a system composed of two (or more) 
subsystems which we know are independent. This means that both the prior 
and the posterior are products. If the subsystem priors are qi{xi) and q2ix2), 
then the prior for the whole system is the product 

q{xi,X2) ^ qi{xi)q2{x2) , (6.8) 

while the joint posterior is constrained within the family 

p{xi,X2) = Pi{xi)p2{x2) . (6.9) 

Further suppose that new information is acquired, say constraints Ci such that 
qi{xi) is updated to Pi{xi), and constraints C2 such that q2ix2) is updated to 
P2{x2)- Axiom 3 is implemented as follows: First we treat the two subsystems 
separately. For subsystem 1 we maximize 

S[pi,q,] = f dx, q,{x,)^ f ^4^) . (6-10) 

subject to constraints Ci on the marginal distribution pi{xi) — J dx2p{xi,X2) 
to select the posterior Pi{xi). The constraints Ci could, for example, include 
normalization, or they could involve the known expected value of a function 
fi{xi), 

J dxifi{xi)pi{xi) ^ J dxidx2 fi{xi)p{xi,X2) ^ Fi . (6.11) 

Similarly, for subsystem 2 we maximize the corresponding S'[p2,'i'2] subject to 
constraints C2 on ^2(2^2) = J dxip{xi,X2) to select the posterior P2{x2)- 

Next the subsystems are treated jointly. Since we are concerned with those 
special examples where we have the information that the subsystems are inde- 
pendent, we are required to search for the posterior within the restricted family 



of joint distributions that take the form of the product (6.9 1; this is an addi- 
tional constraint over and above the original Ci and C2. The new constraint 
p = P1P2 is easily implemented by direct substitution. Instead of maximizing 
the joint entropy, S[p,qiq2], we now maximize 

S[piP2,qiq2] = / dx,dx2 qi{xi)q2{x2)<^ / Pi (^1)^2(3^2) A ^ ^^^^2) 
J \qi{xi)q2{x2) J 

under independent variations 6pi and Sp2 subject to the same constraints Ci and 
C2. The function $ is then determined - or at least constrained - by demanding 
that the selected posterior be Pi{xi)P2{x2)- 

The second point is that the axiom applies to all instances of systems that 
happen to be independent - this is why it is so powerful. The axiom applies to 
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situations where we deal with just two systems - as in the previous paragraph - 
and it also applies when we deal with many, whether just a few or a very large 
number. The axiom applies when the independent subsystems are identical, 
and also when they are not. 

The final conclusion is that probability distributions p{x) should be ranked 
relative to the prior q{x) according to the relative entropy, 

S[p,q]= ~ fdxpix) log^ (6.13) 



The lengthy proof leading to (6.131 is given in the next section. It involves 
three steps. First we show (subsection 6.3.4) that applying Axiom 3 to subsys- 
tems that happen to be identical restricts the entropy functional to a member 
of the one-parameter family of entropies Sr^ [p, q\ parametrized by an "inference 
parameter" ry, 

^"[^''^1 ^ ~ / '^'^P'^"^'') ■ (6.14) 

It is easy to see that there are no singularities for 77 or —1. The limits rj ^ Q 
and 77 — > — 1 are well behaved. In particular, to take 77 ^ use 

y'' = exp(?7 logy) w 1 -f-?7 logy , (6.15) 



which leads to the usual logarithmic entropy, ^oip,?] = S[p,q] given in eq. (6. 13 1. 
Similarly, for r/ ^ — 1 we get S-i[p, q\ = S[q,p\. 

In the second step (subsection 6.3.5) axiom 3 is applied to two independent 
systems that are not identical and could in principle be described by different 
parameters 771 and 772. The consistency demanded by axiom 3 implies that the 
two parameters must be equal, ?7i = 772, and since this must hold for all pairs 
of independent systems we conclude that rj must be a universal constant. In 
the final step the value of this constant - which turns out to be 77 = - is 
determined (subsection 6.3.5) by demanding that axiom 3 apply to N identical 
subsystems where N is very large. 

We can now summarize our overall conclusion: 

The ME method: We want to update from a prior distribution q{x) to a 
posterior distribution p{x) when information in the form of a constraint 
that specifies the allowed posteriors becomes available. The posterior se- 
lected by induction from special cases that implement locality, coordinate 
invariance and consistency for independent subsystems, is that which max- 
imizes the relative entropy S\p, q] subject to the available constraints. No 
interpretation for S[p, q] is given and none is needed. 

This extends the method of maximum entropy beyond its original purpose 
as a rule to assign probabilities from a given underlying measure (MaxEnt) to a 
method for updating probabilities from any arbitrary prior (ME). Furthermore, 
the logic behind the updating procedure does not rely on any particular meaning 
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assigned to the entropy, either in terms of information, or heat, or disorder. 
Entropy is merely a tool for inductive inference; we do not need to know what 
it means; we only need to know how to use it. 

The derivation above has singled out a unique S[p, q] to be used in inductive 
inference. Other 'entropies' could turn out to be useful for other purposes - 
perhaps as measures of information, or of ecological diversity, or something else 
- but they are not an induction from the special cases set down in the axioms. 



6.3 The proofs 

In this section we establish the consequences of the three axioms leading to the 



final result eq.(6.13). The details of the proofs are important not just because 
they lead to our final conclusions, but also because the translation of the verbal 
statement of the axioms into precise mathematical form is a crucial part of 
unambiguously specifying what the axioms actually say. 

6.3.1 Axiom 1: Locality 



Here we prove that axiom 1 leads to the expression eq.(6.1) for S[p,q]. The 
requirement that probabilities be normalized is handled by imposing normaliza- 
tion as one among so many other constraints that one might wish to impose. To 
simplify the proof we consider the case of a discrete variable, pi with i = 1 . . . n, 
so that S[p,q] — S{pi . . .pn,qi ■ ■ The generalization to a continuum is 

straightforward. 

Suppose the space of states X is partitioned into two non-overlapping do- 
mains D and V with VWD' = X, and that the information to be processed is 
in the form of a constraint that refers to the domain 2?' , 

E ajpj = A . (6.16) 

Axiom 1 states that the constraint on V does not have an influence on the 
conditional probabilities Pi\xi- It may however influence the probabilities pi 
within V through an overall multiplicative factor. To deal with this complication 
consider then a special case where the overall probabilities of V and V are 
constrained too, 

Y^Pt^Pv and J2 Pj ^ P-D' , (6.17) 
iev jev 

with Pj) + P-D' = 1- Under these special circumstances constraints on V will 
not influence piS within 2?, and vice versa. 

To obtain the posterior maximize S[p, q] subject to these three constraints, 

Q=\5S-\(Y.P^-Pv 

Kiev 

- X' I J2P^-Pv']+^^l E ajp, - A 
\jev' J \jev' . 
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leading to 

dS 



^ = A for ieV , (6.18) 
dpi 

dS 

— = A' + ^flj for j eV . (6.19) 



Eqs.( 6.16|6.19 l are n + 3 equations we must solve for the piS and the three 

S{pi . . . Pn, Qi . . . Qn) its derivative 

Mpi . . .Pn,qi ■ ■ -Qn) (6.20) 



Lagrange multipliers. Since S = S{pi . . . Pn, Qi . . . Qn) its derivative 

dS 



dpi 

could in principle also depend on all 2n variables. But this violates the locality 
axiom because any arbitrary change in Oj within T)' would influence the piS 
within T). The only way that probabilities within T) can be shielded from arbi- 
trary changes in the constraints pertaining to T)' is that the functions fi with 
i £V depend only on piS while the functions fj depend only on pjS. Further- 
more, this must hold not just for one particular partition of X into domains 
T) and T)' , it must hold for all conceivable partitions. Therefore fi can depend 
only on pi and, at this point, on any of the gs, 

dS 

= fi{Pi,qi ■■■qn) ■ (6.21) 

OPi 

But the power of the locality axiom is not exhausted yet. The information 
to be incorporated into the posterior can enter not just through constraints but 
also through the prior. Suppose that the local information about domain V is 
altered by changing the prior within V . Let qj — > qj + Sqj for j e V . Then 



(6.21 ) becomes 

dS 

^ = fi{Pi, qi...qj+ Sqj ...qn) (6.22) 

OPi 

which shows that pi with i Cz V will be influenced by information about V 
unless fi with z g I? is independent of all the qjs for j £ V . Again, this must 
hold for all partitions into V and V , and therefore, 

dS 

-— = f^{p^,q^) for all leX. (6.23) 

OPi 

Integrating, one obtains 

S[p, q] = Y.F.i{p^, qi) + constant . (6.24) 



for some undetermined functions Fi . The corresponding expression for a contin- 
uous variable x is obtained replacing i by x, and the sum over i by an integral 
over X leading to eq.(6.1 1. 
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6.3.2 Axiom 2: Coordinate invariance 



transforms as a density and rewrite the expression (6.11 for the entropy in the 
form 



Next we prove eq.(6.6l It is convenient to introduce a function m{x) which 

be exp 

p{x) 



1 



S[p,q] = I dxm{x)^^F [ ^^^m{x), —^m{x),x 
m[x) \m[x) mix) 



P{x) q{x) 
^m{x) ' m{x) 

where the function $ is defined by 



^Jdxmi^)<, 



,m{x),x , 



<I>(a, P, m, x) '= — F(am, /3m, m, x). 
m 



(6.25) 
(6.26) 

(6.27) 



Next, we consider a special situation where the new information are con- 
straints which do not favor one coordinate system over another. For example 
consider the constraint 

(6.28) 



(6.29) 



dxp{x)a{x) = A 
where a{x) is a scalar, i.e., invariant under coordinate changes, 

a{x) — > a'{x') — a(x). 



The usual normalization condition J dxp{x) = 1 is a simple example of a scalar 
constraint. 

Maximizing S[p, q] subject to the constraint. 



S[p,q] + X^J dxp{x)a{x) 



= 0, 



(6.30) 



gives 



p{x) q{x) , 
$ ( -^-V4, ^■V4,m,(a;),x = Xa{x) , 



m(x) ' m(x) 

where the dot represents the derivative with respect to the first argument 

dcf d^{a,P,m,x) 



(6.31) 



<& (a, /3, TO, x) 



da 



But we could have started using the primed coordinates. 



— — , ———,m'{x'),x' = X'a'{x') 
'{x') myx') J 



or, using (6.3 1 and ( 6.29[ ), 



<^\^y^ym{xUx'),x']=\'a{x). 
myx) m'yx) ' 



(6.32) 



(6.33) 



(6.34) 
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Dividing (6.34) by (6.311 we get 



4 {a, (3,mj,x') _ X' 
3> (a, m, a;) A 



(6.35) 



This identity should hold for any transformation x = r(a;'). On the right hand 
side the multipliers A and A' are just constants; the ratio A'/A might depend on 
the transformation T but it does not depend on x. Consider the special case 
of a transformation T that has unit determinant everywhere, 7 = 1, and differs 
from the identity transformation only within some arbitrary region V. Since for 



X outside this region V we have x ^ x' , the left hand side of eq.(6.35) equals 
1. Thus, for this particular T the ratio is A'/A = 1; but A'/A = constant, so 
A'/A = 1 holds within T) as well. Therefore, for x within V, 



$ (a, f3, TO, x') = $ {a, l3, to, x) 



(6.36) 



Since the choice of D is arbitrary we conclude is that the function <i> cannot 
depend on its third argument, <I> = $ {a,/3,m). 



Having eliminated the third argument, let us go back to eq.(6.35) 



<I> {a,(3,m-f) _ A' 



i> {a, (3,711) 



A 



(6.37) 



and consider a different transformation F, one with unit determinant 7=1 only 
outside the region V. Therefore the constant ratio A'/A is again equal to 1, so 
that 

$(q;,/3,to7) = $(a,/3,TO) . (6.38) 

But within D the transformation F is quite arbitrary, it could have any arbitrary 
Jacobian 7 1. Therefore the function $ cannot depend on its second argument 
either, and therefore <i> = <i>(a,/3). Integrating with respect to a gives <& = 
<&(«, (3) + constant. The additive constant, which could depend on /3, has no 
effect on the maximization and can be dropped. This completes the proof of 
eq.(|6.6|). 



6.3.3 Axiom 1 again 

The locality axiom implies that when there are no constraints the selected pos- 
terior distribution should coincide with the prior distribution. This provides 
us with an interpretation of the density m{x) that had been artificially intro- 



duced. The argument is simple: maximize S[p,q] in (6.6 1 subject to the single 
requirement of normalization. 



to get 



S[p, q] + \ [ / dxp{x) - 1 



= 0, 



p{x) q{x) 



m{x) m{x) 



(6.39) 



(6.40) 
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Since A is a constant, the left hand side must be independent of x for arbitrary 
choices of the prior q{x). This could, for example, be accomplished if the func- 
tion <I>(q;,/3) were itself a constant, independent of its arguments a and f3. But 
this gives 

$(a,/3) = cia + C2 (6.41) 

where Ci and C2 are constants and leads to the unacceptable form S[p, q] oc 
J dxp{x) + constant. 

If the dependence on x cannot be eliminated by an appropriate choice of <&, 



we must secure it by a choice of m(x). Eq.(6.40) is an equation for p{x). In 



the absence of new information the selected posterior distribution must coincide 



with the prior, P{x) = q{x). The obvious way to secure that (6.401 be indepen- 
dent of X is to choose m{x) oc q{x). Therefore m{x) must, except for an overall 
normalization, be chosen to coincide with the prior distribution. 

6.3.4 Axiom 3: Consistency for identical independent sub- 
systems 

In this subsection we show that applying axiom 3 to subsystems that happen to 
be identical restricts the entropy functional to a member of the one-parameter 
family of ry-entropies S',,[p, q] parametrized by rj. For 77 = one obtains the 



standard logarithmic entropy, eq.(6.13) 



So[p, l] = -J dxp{x) log ^ . (6.42) 

For T] = —1 one obtains 

S^l[p,q]^ f dxq{x)logP^ , (6.43) 

which coincides with So[q,p\ with the arguments switched. Finally, for a generic 
value of 7] 7^ — 1 , the result is 

S,[p,q] = -ldxp{x)(^^J . (6.44) 

It is worthwhile to recall that the objective of this whole exercise is to 
rank probability distributions according to preference and therefore different 
entropies that induce the same ranking scheme are effectively equivalent. This 
is very convenient as it allows considerable simplifications by an appropriate 
choice of additive and multiplicative constants. Taking advantage of this free- 



dom we can, for example, combine the three expressions (6.421, (6.43), and 



(6.44|) into the single expression 



S,[p, q] = — ^ ( 1 - / dxp^'+\-^ ] , (6.45) 



that we met earlier in cq.(6.14l. 
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The proof below is fairly lengthy and may be skipped on a first reading. 
It follows the treatment in [Caticha Giffin 06] and is based upon and extends 
a previous proof by Karbelkar who showed that belonging to the family of rj- 
entropies is a sufficient condition to satisfy the consistency axiom for identical 
systems. He conjectured but did not prove that this was perhaps also a necessary 
condition. [Karbelkar 86] Although necessity was not essential to his argument 
it is crucial for ours. We show below that for identical subsystems there are no 
acceptable entropies outside the S*^ family. 

First we treat the subsystems separately. For subsystem 1 we maximize the 
entropy S[pi,qi] subject to normalization and the constraint Ci in eq.(6.11l. 
Introduce Lagrange multipliers ai and Ai, 



dxi Pi - 1 



0: 



(6.46) 



which gives 



Mxi) 
qi{xi) 



= Ai/i(a;i) + ai 



(6.47) 



where the prime indicates a derivative with respect to the argument, ^'{y) = 
d^{y)/dy. For subsystem 2 we need only consider the extreme situation where 
the constraints C2 determine the posterior completely: P2{x2) — P2{x2). 

Next we treat the subsystems jointly. The constraints C2 are easily imple- 
mented by direct substitution and thus, we maximize the entropy S\piP2, 9192] 
by varying over pi subject to normalization and the constraint Ci in eq.(6.11l. 
Introduce Lagrange multipliers a and A, 



5[piP2, 9192] - A / dxifipi - Fi 



dxi pi — 1 



0, (6.48) 



which gives 



dx2P2<^'(—] = A[P2,92]/i(a:i) + a[P2,92] , (6.49) 
V9192 



where the multipliers A and a are independent of xi but could in principle be 
functionals of P2 and q2. 

The consistency condition that constrains the form of <f> is that if the solution 
to eq.(6.47l is Pi{xi) then the solution to eq.( 6.49[ ) must also be Pi{xi), and 
this must be true irrespective of the choice of P2(a^2)- Let us then consider 
a small change P2 ^ P2 + 5P2 that preserves the normalization of P2. First 



introduce a Lagrange multiplier a2 and rewrite eq.(6.49l as 



dx2 P2<^' 



P1P2 
91 92 



-a2 



dx2Pp2 - 1 



A[P2,92]/i(2:i)+a[P2,g2] , (6.50) 



where we have replaced pi by the known solution Pi and thereby effectively 
transformed eqs. (6.471 and (6.49) into an equation for The 6P2 variation 
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gives, 



P1P2 



P1P2 

91 92 



P1P2 

91 92 



02 



(6.51) 



Next use eq. (6.471 to eliminate fi{xi), 



where 



P1P2 
91 92 



P1P2 
91 92 



P1P2 
91 92 



A[P2,92]*' 



i?[P2,92] , (6.52) 



^[^^2,92] 



1 SX 

Y1JP2 



and B[P2,92] 



5X ai 

mx~i 



Sa 



Oi2 , 



(6.53) 



are at this point unknown functionals of P2 and q2. Differentiating eq.(6.52l 
with respect to xi the B term drops out and we get 



^[-P2,92] = 



dxi 



d 
dxi 



P1P2 
91 92 



P1P2 
91 92 



<j)" 



P1P2 
91 92 



(6.54) 



which shows that A is not a functional of P2 and g2 but a mere function of P2 /92 ■ 
Substituting back into eq.(6.52l we see that the same is true for B. Therefore 
eq.(6.52l can be written as 

^' (2/12/2) + 2/12/2*" (2/12/2) - ^(2/2)*' (2/1) + 5(^2) , (6.55) 

where yi — Pi/91, 2/2 = P2/92, and ^(2/2), ^(2/2) are unknown functions of j/2- 
Now we specialize to identical subsystems. Then we can exchange the labels 
1^2, and we get 

A{y2W (2/1) + 5(2/2) = ^(2/1)*' (2/2) + B{y{) . (6.56) 
To find the unknown functions A and B differentiate with respect to 2/2, 



A'{y2W {yi) + B'{y2)^A{y,)<^" (ya) 
and then with respect to yi to get 

A'{yi) A'{y2) 



Integrate to get 



(2/1) $" (2/2) 



A(2;i) = aa>' (2/1) 



const 



then substitute back into eq.(6.57) and integrate again to get 
P'(2/2) = (2/2) and 5(2/2) = 6$' (2/2) + c 



(6.57) 

(6.58) 
(6.59) 
(6.60) 



where h and c are constants. We can check that A{y) and B{y) are indeed 
solutions of eq. (6.561. Substituting into eq.(6.55l gives 

(2/12/2) + 2/i2/2«'" (2/12/2) = a*' (2/1) (2/2) + h (zji) + (2/2)] + c . (6.61) 



6.3. THE PROOFS 



131 



This is a peculiar difFerential equation. We can think of it as one differential 
equation for 4>' (yi) for each given constant value of j/2 but there is a complication 
in that the various (constant) coefficients <&' (2/2) are themselves unknown. To 
solve for <I> choose a fixed value of j/2 , say y2 — I, 



(y) - 7,$' (y) - K = , 



(6.62) 



where rj — a^' (1) + 6 — 1 and k — b^' (1) + c. To eliminate the constant k 
differentiate with respect to y, 



2/$"' + (1 - 7?) = , 

which is a linear homogeneous equation and is easy to integrate. 
For generic values of 77 7^ — 1 , the solution is 



$"(2/) cx y 



rj-l 



<i>'(2/) = ay" + /3 



(6.63) 



(6.64) 



The constants a and (3 are chosen so that this is a solution of eq.(6.61l for all 
values of 7/2 (and not just for y2 — 1). Substituting into eq.(6.61 1 and equating 
the coefficients of various powers of yiy2, yi, and 2/2 gives three conditions on 
the two constants a and /3, 



a(l + ?/) = aa^, = aaf3 + ba, /3 ^ af]"^ + 2b(3 + c 



(6.65) 



The nontrivial {a ^ 0) solutions are a — (1 + 7y)/a and /3 = —b/a, while the 
third equation gives c — 6(1 — b)/4a. We conclude that for generic values of 77 
the solution of eq.(6.61 1 is 



'i>(2/)--y''+'-72/ + ^ 



b 

—1 
a" 



(6.66) 



where C is a new constant. Substituting into eq.(6.7) yields 

S4p,q] - dxp{x) (^^y -^J dxp{x) + C J dxq{x) . (6.67) 

This complicated expression can be simplified considerably by exploiting the 
freedom to choose additive and multiplicative constants. We can drop the last 
two terms and choose a = —1 so that the preferred distribution is that which 
maximizes entropy. This reproduces eq.(6.44l. 



For 7/ = we return to eq.(6.63l and integrate twice to get 
*(y) = a' ylogy + b' y + c' , 



for some new constants a', 6', and c'. Substituting into eq.(6.7) yields 



So[p,q] ^ a! I dxp{x)\o. 



+ b' I dxp{x)+c' / dxq{x) . 



(6.68) 



(6.69) 
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Again, choosing a' = —1 and dropping the last two terms does not affect 
the ranking scheme. This yields the standard expression for relative entropy, 
eq.(|6.42|). 



Finally, for r] = —1 integrating eq.(6.63l twice gives 
^y) = a"logy + b"y + c" , 



(6.70) 



for some new constants a", 6", and c". Substituting into eq.(6.7l yields 

So[p, q] = a" J dx q{x) log ^ + b" J dxp{x) + c" J dx q{x) . (6.71) 



Again, choosing a" = 1 and dropping the last two terms yields eq.(6.43l. This 
completes our derivation. 



6.3.5 Axiom 3: Consistency for non-identical subsystems 

Let us summarize our results so far. The goal is to update probabilities by rank- 
ing the distributions according to an entropy S that is of general applicability. 
The allowed functional forms of the entropy S have been constrained down to a 
member of the one-dimensional family Srj. One might be tempted to conclude 
that there is no S of universal applicability; that inferences about different sys- 
tems could to be carried out with different 77-entropies. But we have not yet 
exhausted the full power of the consistency axiom 3. Consistency is universally 
desirable; there is no reason why it should be limited to identical systems. 

To proceed further we ask: What is ij? Is it a property of the individual 
carrying out the inference or of the system under investigation? The former is 
unacceptable; we insist that the updating must be objective in that different 
individuals with the same prior and with the same constraints must make the 
same inference. Therefore the "inference parameter" 77 can only be a property 
of the system. 

Consider two different systems characterized by ryi and 772- Let us further 
suppose that these systems are known to be independent (perhaps system #1 
lives here on Earth while system #2 lives in a distant galaxy) so that they fall 
under the jurisdiction of axiom 3. Separate inferences about systems #1 and #2 
are carried out with S'^Jpijqi] and 5*^2 [P2, 92] respectively. For the combined 
system we are also required to use an 77-entropy, say Srj[piP2, 9192] • The question 
is what rjs do we choose that will lead to consistent inferences whether we treat 
the systems separately or jointly. The results of the previous subsection indicate 
that a joint inference with S'^[piP2, 9192] is equivalent to separate inferences with 
Sriipi, qi] and Sri[p2, q2\- Therefore we must choose 77 = rji and also r/ = 772 which 
is possible only if we had rji = 772 from the start. 

But this is not all: consider a third system #3 that also lives here on Earth. 
We do not know whether system #3 is independent from system #1 or not but 
we can confidently assert that it will certainly be independent of the system 
=f/=2 living in the distant galaxy. The argument of the previous paragraph leads 
us to conclude that 773 ~ 772, and therefore that 773 = 771 even when systems 
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#1 and #3 are not known to be independent! We conclude that all systems 
must be characterized by the same parameter rj whether they are independent or 
not because we can always find a common reference system that is sufficiently 
distant to be independent of any two of them. The inference parameter 77 is a 
universal constant, the value of which is at this point still unknown. 

The power of a consistency argument resides in its universal applicability: 
if an entropy S[p, q] exists then it must be one chosen from among the 5'^[p, q\. 
The remaining problem is to determine this universal constant 77. Here we give 
one argument; in the next subsection we give another one. 

One possibility is to regard 77 as a quantity to be determined experimen- 
tally. Are there systems for which inferences based on a known value of rj have 
repeatedly led to success? The answer is yes; they are quite common. 

As we discussed in Chapter 5 statistical mechanics and thus thermodynamics 
are theories of inference based on the value 77 = 0. The relevant entropy, which 
is the Boltzmann-Gibbs-Shannon entropy, can be interpreted as the special case 
of the ME when one updates from a uniform prior. It is an experimental fact 
without any known exceptions that inferences about all physical, chemical and 
biological systems that are in thermal equilibrium or close to it can be carried 
out by assuming that 77 = 0. Let us emphasize that this is not an obscure and 
rare example of purely academic interest; these systems comprise essentially all 
of natural science. (Included is every instance where it is useful to introduce a 
notion of temperature.) 

In conclusion: consistency for non-identical systems requires that 7; be a 
universal constant and there is abundant experimental evidence for its value 
being 7; = 0. Other 7;-entropies may turn out to be useful for other purposes but 
the logarithmic entropy S[p,q] in eq.(6.1S) provides the only consistent ranking 
criterion for updating probabilities that can claim general applicability. 



6.3.6 Axiom 3: Consistency with the law of large numbers 

Here we offer a second argument, also based on a broader application of axiom 3, 
that the value of the universal constant rj must be 7; = 0. We require consistency 
for large numbers of independent identical subsystems. In such cases the weak 
law of large numbers is sufficient to make the desired inferences. 

Let the state for each individual system be described by a discrete variable 

7 = 1 . . . 77^. 

First we treat the individual systems separately. The identical priors for 
the individual systems are qi and the available information is that the potential 
posteriors pi are subject, for example, to an expectation value constraint such 
as (a) = A, where A is some specified value and (a) — '^aiPi. The preferred 
posterior Pi is found maximizing the 77-entropy S',Jp, q] subject to (a) = A. 

To treat the systems jointly we let the number of systems found in state i 
be Hi, and let fi — rii/N be the corresponding frequency. The two descriptions 
are related by the law of large numbers: for large N the frequencies fi converge 
(in probability) to the desired posterior Pi while the sample average d — ^aifi 
converges (also in probability) to the expected value (a) — A. 
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Now we consider the set of N systems treated jointly. The probabihty of 
a particular frequency distribution / — (/i . . . /«) generated by the prior q is 
given by the multinomial distribution, 



Nif\q}^Zrr^ with Zn^ = N. (6.72) 



m 

ni\...n,n\ ' i~i 
When the rii are sufficiently large we can use Stirling's approximation, 

log n\ = n log n — n + log + 0(1 /n) . (6.73) 

Then 

loggAT (/|g) w7VlogA^-7V + logV2^ 

- Yl, log m "111 + log y/2TTn, - Hi log Qi) 

i 

= -^E I log ^ - E log - ( - 1) log 

= iV5[/,g]-Elogv^-(A^-l)logV2^, (6.74) 



where S[f,q\ is the 77 = entropy given by eq.(6.13). Therefore for large TV can 
be written as 

Qn (/k) « CNiW^r^'^ exp(iV5[/, q]) (6.75) 

i 

where Cjv is a normalization constant. The Gibbs inequality S[f,q] < 0, 



eq.(4.21 1, shows that for large N the probability Qn {f\<l) shows an exceedingly 
sharp peak. The most likely frequency distribution is numerically equal to the 
probability distribution g^. This is the weak law of large numbers. Equivalently, 
we can rewrite it as 

j^logQNif\q)~S[f,q] + rN , (6.76) 

where rjv is a correction that vanishes as iV 00. This means that finding the 
most probable frequency distribution is equivalent to maximizing the entropy 
S[f,q]. 

The most probable frequency distribution that satisfies the constraint a ~ A 
is the distribution that maximizes Qn (/k) subject to the constraint a ^ A, 
which is equivalent to maximizing the entropy S[f, q] subject to a = A. In 
the limit of large the frequencies fi converge (in probability) to the desired 
posterior Pi while the sample average a = '^aifi converges (also in probability) 
to the expected value (a) = A. 

The two procedures agree only when we choose 77 = 0. Inferences carried 
out with with 7] =/= arc not consistent with inferences from the law of large 
numbers. This is the Principle of Eliminative Induction in action: it is the 
successful falsification of all rival T^-entropies that corroborates the surviving 
entropy with rj = Q. The reason the competing 77-cntropies are discarded is 
clear: 77 7^ is inconsistent with the law of large numbers. 
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[Csiszar 84] and [Grcndar 01] have argued that the asymptotic argument 
above provides by itself a valid justification for the ME method of updating. An 
agent whose prior is q receives the information (a) = A which can be reasonably 
interpreted as a sample average a = A over a large ensemble of N trials. The 
agent's beliefs are updated so that the posterior P coincides with the most 
probable / distribution. This is quite compelling but, of course, as a justification 
of the ME method it is restricted to situations where it is natural to think in 
terms of ensembles with large N. This justification is not nearly as compelling 
for singular events for which large ensembles either do not exist or are too 
unnatural and contrived. From our point of view the asymptotic argument 
above does not by itself provide a fully convincing justification for the universal 
validity of the ME method but it does provide considerable inductive support. 
It serves as a valuable consistency check that must be passed by any inductive 
inference procedure that claims to be of general applicability. 

6.4 Random remarks 

6.4.1 On deductive vs. inductive systems 

In a deductive axiomatic system certain statements arc chosen as axioms and 
other statements called theorems are derived from them. The theorems can be 
asserted to be true only when conditions are such that the axioms hold true. 
Within a deductive axiomatic system it makes no sense to make assertions that 
go beyond the reach of applicability of the axioms. In contrast the purpose of 
eliminative induction is precisely to venture into regions beyond those known 
special cases the axioms and accordingly, the truth of the resulting inferences 
- the theorems - is not guaranteed. 

A second interesting difference is that in a deductive system there is a certain 
preference for minimizing the number of axioms as this clarifies the relations 
among various elements of the system and the structure of the whole. In contrast 
when doing induction one strives to maximize the number of axioms as it is much 
safer to induce from many known instances than from just a few. 

6.4.2 On priors 

All entropies are relative entropies. In the case of a discrete variable, if one 
assigns equal a priori probabilities, qi = 1, one obtains the Boltzmann-Gibbs- 
Shannon entropy, S[p] = — X^i Pi^'^&Pi ■ The notation S[p] has a serious draw- 
back: it misleads one into thinking that S depends on p only. In particular, 
we emphasize that whenever S[p] is used, the prior measure qi = 1 has been 
implicitly assumed. In Shannon's axioms, for example, this choice is implicitly 
made in his first axiom, when he states that the entropy is a function of the 
probabilities S = S{pi...pn) and nothing else, and also in his second axiom when 
the uniform distribution p^ = 1/n is singled out for special treatment. 

The absence of an explicit reference to a prior q^ may erroneously suggest 
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that prior distributions have been rendered unnecessary and can be ehminated. 
It suggests that it is possible to transform information {i.e., constraints) di- 
rectly into posterior distributions in a totally objective and unique way. This 
was Jaynes' hope for the MaxEnt program. If this were true the old controversy, 
of whether probabilities are subjective or objective, would have been resolved - 
probabilities would ultimately be totally objective. But the prior qt = 1 is im- 
plicit in S[p\; the postulate of equal a priori probabilities or Laplace's "Principle 
of Insufficient Reason" still plays a major, though perhaps hidden, role. Any 
claims that probabilities assigned using maximum entropy will yield absolutely 
objective results are unfounded; not all subjectivity has been eliminated. Just as 
with Hayes' theorem, what is objective here is the manner in which information 
is processed to update from a prior to a posterior, and not the prior probabilities 
themselves. 

Choosing the prior density q{x) can be tricky. Sometimes symmetry consid- 
erations can be useful in fixing the prior (three examples were given in section 
4.5) but otherwise there is no fixed set of rules to translate information into 
a probability distribution except, of course, for Bayes' theorem and the ME 
method themselves. 

What if the prior q(x) vanishes for some values of x7 S[p, q] can be infinitely 
negative when q{x) vanishes within some region V. In other words, the ME 
method confers an overwhelming preference on those distributions p{x) that 
vanish whenever q(x) does. One must emphasize that this is as it shoTild be; 
it is not a problem. A similar situation also arises in the context of Bayes' 
theorem where a vanishing prior represents a tremendously serious commitment 
because no amount of data to the contrary would allow us to revise it. In both 
ME and Bayes updating we should recognize the implications of assigning a 
vanishing prior. Assigning a very low but non-zero prior represents a safer and 
less prejudiced representation of one's beliefs. 

For more on the choice of priors see the review [Kass Wasserman 96]; in 
particular for entropic priors see [Rodriguez 90-03, Caticha Preuss 04] 

6.4.3 Comments on other axiomatizations 

One feature that distinguishes the axiomatizations proposed by various authors 
is how they justify maximizing a functional. In other words, why maximum 
entropy? In the approach of Shore and Johnson this question receives no answer; 
it is just one of the axioms. Csiszar provides a better answer. He derives the 
'maximize a functional' rule from reasonable axioms of regularity and locality 
[Csiszar 91]. In SkilHng's and in the approach developed here the rule is not 
derived, but it does not go unexplained either: it is imposed by design, it is 
justified by the function that S is supposed to perform, to achieve a transitive 
ranking. 

Both Shore and Johnson and Csiszar require, and it is not clear why, that 
updating from a prior must lead to a unique posterior, and accordingly, there 
is a restriction that the constraints define a convex set. In Skilling's approach 
and in the one advocated here there is no requirement of uniqueness, we are 
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perfectly willing to entertain situations where the available information points 
to several equally preferable distributions. 

There is another important difference between the axiomatic approach pre- 
sented by Csiszar and the present one. Since our ME method is a method 
for induction we are justified in applying the method as if it were of universal 
applicability. As with all inductive procedures, in any particular instance of 
induction can turn out to be wrong because, for example, not all relevant 
information has been taken into account - but this does not change the fact 
that ME is still the unique inductive inference method that generalizes from the 
special cases chosen as axioms. Csiszar's version of the MaxEnt method is not 
designed to generalize beyond the axioms. His method was developed for linear 
constraints and therefore he does not feel justified in carrying out his deductions 
beyond the cases of linear constraints. In our case, the application to non-linear 
constraints is precisely the kind of induction the ME method was designed to 
perform. 

It is interesting that if instead of axiomatizing the inference process, one 
axiomatizes the entropy itself by specifying those properties expected of a mea- 
sure of separation between (possibly unnormalized) distributions one is led to a 
continuum of ?7-entropies, [Amari 85] 

^"t^' " rj{v+ 1) / '''' ^^"^ ^ ~ ''^ ~ ' ^^-^^^ 

labelled by a parameter r). These entropies arc equivalent, for the purpose of 
updating, to the relative Renyi entropies [Renyi 61, Aczel 75]. The shortcoming 
of this approach is that it is not clear when and how such entropies are to 
be used, which features of a probability distribution arc being updated and 
which preserved, or even in what sense do these entropies measure an amount 
of information. Remarkably, if one further requires that 5^ be additive over 
independent sources of uncertainty, as any self-respecting measure ought to be, 
then the continuum in r] is restricted to just the two values 77 = and 77 = —1 
which correspond to the entropies S\p,q] and S[q,p]. 

For the special case when p is normalized and a uniform prior q = 1 we get 
(dropping the integral over q) 




(6.78) 



A related entropy 

5; = i (^1 - I dxp^+'^ (6.79) 

has been proposed in [Tsallis 88] and forms the foundation of his non-extensive 
statistical mechanics. Clearly these two entropies are equivalent in that they 
generate equivalent variational problems - maximizing Srj is equivalent to maxi- 
mizing S'^. To conclude our brief remarks on the entropies S*,, we point out that 
quite apart from the diHiculty of achieving consistency with the law of large 
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numbers, some the probability distributions obtained maximizing S*,, may also 
be derived through a more standard use of MaxEnt or ME as advocated in these 
lectures. [Plastino 94] 



6.5 Bayes' rule as a special case of ME 

Since the ME method and Bayes' rule are both designed for updating proba- 
bilities, and both invoke a Principle of Minimal Updating, it is important to 
explore the relations between them. In particular we would like to know if the 
two are mutually consistent or not. [Caticha Giffin 06] 

As described in section 2.10 the goal is to update our beliefs about 9 ^ Q {6 
represents one or many parameters) on the basis of three pieces of information: 
(1) the prior information codified into a prior distribution q{9); (2) the data 
X € X (obtained in one or many experiments); and (3) the known relation 
between 9 and x given by the model as defined by the sampling distribution 
or likelihood, q{x\9). The updating consists of replacing the prior probability 
distribution q(9) by a posterior distribution P{9) that applies after the data has 
been processed. 

The crucial element that will allow Bayes' rule to be smoothly incorporated 
into the ME scheme is the realization that before the data information is avail- 
able not only we do not know 9, we do not know x either. Thus, the relevant 
space for inference is not Q but the product space Q x X and the relevant joint 
prior is q{x,9) = q{9)q{x\9). We should emphasize that the information about 
how X is related to 9 is contained in the functional form of the distribution q{x\9) 
- for example, whether it is a Gaussian or a Cauchy distribution or something 
else - and not in the actual values of the arguments x and 9 which are, at this 
point, still unknown. 

Next we collect data and the observed values turn out to be X. We must 
update to a posterior that lies within the family of distributions p{x, 9) that 
reflect the fact that x is now known, 

p(x) = J d9p{9, x) = 5{x - X) . (6.80) 

This data information constrains but is not sufficient to determine the joint 
distribution 

p(x, 9) = p{x)p{9\x) = 6{x - X)p{9\X) . (6.81) 

Any choice of p(9\X^ is in principle possible. So far the formulation of the prob- 
lem parallels section 2.10 exactly. We are, after all, solving the same problem. 
Next we apply the ME method and show that we get the same answer. 

According to the ME method the selected joint posterior P{x,9) is that 
which maximizes the entropy, 

S[p, q] = -Jdxd9 p{x, 9) log 4^ , (6.82) 

q{x^ 9) 

subject to the appropriate constraints. Note that the information in the data. 



eq.(6.80l, represents an infinite number of constraints on the family p{x,( 
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for each value of x there is one constraint and one Lagrange multipHer A(x). 
Maximizing S, ( |6.82 1, subject to ( |6.80[ ) and normalization, 



S{S + a [Jdxde p{x, 0)-l]+ Jdx X{x) [Jd0 p{x, 0) ~ S{x ~ X)]} ^ , 

(6.83) 

yields the joint posterior, 

P{x,0) = q{x,0)^ , (6.84) 
where Z is a normalization constant, and the multiplier X{x) is determined from 



(6.80) 



X{x) X{x) 

Jd0q{x,0)—=qix)— = S{x^X), (6.85) 
so that the joint posterior is 

P{x, 9) = q{x, 0) = 5{x - X)q{0\x) , (6.86) 
q{x) 

The corresponding marginal posterior probability P{d) is 

P{0) = JdxP{9,x) = q{0\X) ^ q{9f-^ , (6.87) 



which is recognized as Bayes' rule, eq.( 2.123| . Thus Bayes' rule is consistent 



with, and indeed, is a special case of the ME method. 

To summarize: the prior q(x,9) = q{x)q{0\x) is updated to the posterior 
P{x,0) = P{x)P{9\x) where P{x) = d{x — X) is fixed by the observed data 
while P{0\X) ~ q{9\X) remains unchanged. Note that in accordance with the 
philosophy that drives the ME method one only updates those aspects of one 's 
beliefs for which corrective new evidence has been supplied. 

I conclude with a few simple examples that show how the ME allows gener- 
alizations of Bayes' rule. The background for these generalized Bayes problems 
is the familiar one: We want to make inferences about some variables 9 on the 
basis of information about other variables x. As before, the prior information 
consists of our prior knowledge about given by the distribution q{0) and the 
relation between x and is given by the likelihood q(x\0); thus, the prior joint 
distribution q(x, 0) is known. But now the information about x is much more 
limited. 



Bayes updating with uncertain data 

The data is uncertain: x is not known. The marginal posterior p{x) is no longer 
a sharp delta function but some other known distribution, p{x) = Po{x). This 
is still an infinite number of constraints 



p{x) = Jd0p{9,x) = PD{x) , 



(6.88) 
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that are easily handled by ME. Maximizing S, (6.821, subject to (6.881 and 
normalization, leads to 

P{x,9) = PD{x)q{9\x) . (6.89) 
The corresponding marginal posterior, 

P{0) = JdxPD{x)q{0\x) = q{e)JdxPD{x)^^^ , (6.90) 

q(x) 

is known as Jeffrey's rule which we met earlier in section 2.10. 

Bayes updating with information about x moments 

Now we have even less information: p{x) is not known. All we know about p{x) 
is an expected value 

(/) - Jdxpix)f{x) = F . (6.91) 
Maximizing S, ( |6.82| , subject to ( |6.91[ ) and normalization, 

S{S + a[J dxdO p{x, 61) - 1] + A/ dxdO p{x, e)f{x) - F] ^ Q , (6.92) 

yields the joint posterior, 

p{x,e) ^ q{x,e) , (6.93) 

where the normalization constant Z and the multiplier A are obtained from 

Z = /dx <z(x)e^-^(") and '^^^ ^ F . (6.94) 
The corresponding marginal posterior is 

p{e) = q{e)Sdx -^q{x\e) . (6.95) 

These two examples (6.901 and ( |6.95| are sufficiently intuitive that one could 
have written them down directly without deploying the full machinery of the ME 
method, but they do serve to illustrate the essential compatibility of Bayesian 
and Maximum Entropy methods. Next we consider a slightly less trivial exam- 
ple. 

Updating with data and information about 9 moments 

Here we follow [Giffin Caticha 07]. In addition to data about x we have addi- 
tional information about 9 in the form of a constraint on the expected value of 
some function f{9), 

Jdxd9 P{x, 9)f{9) = {f{9)) = F . (6.96) 



In the standard Bayesian practice it is possible to impose constraint infor- 
mation at the level of the prior, but this information need not be preserved 
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in the posterior. What we do here that differs from the standard Bayes' rule 
is that we can require that the constraint (6.961 be satisfied by the posterior 
distribution. 

Maximizing the entropy (6.821 subject to normahzation, the data constraint 

(6.97) 



(6.80), and the moment constraint (6.961 yields the joint posterior 

gA(x)+/3/(9) 



Pix,e) = q{x,e) 

where z is a normalization constant, 

z = Jdxde e^(")+^^(^)q(x, 9) . (6.98) 
The Lagrange multipliers A(a;) are determined from the data constraint, (??), 



z Zq{X) 
so that the joint posterior becomes 



where X) = Jd9 ef^^^'^>q{e\X) , 



P{x,9) = S{x^X)q{9\X)- 



p/3/(e) 



Z 



(6.99) 



(6.100) 



The remaining Lagrange multiplier (3 is determined by imposing that the pos- 
terior P{x,9) satisfy the constraint (6.961. This yields an implicit equation for 
/3, 



aiogz 

8(3 



= F . 



(6.101) 



Note that since Z = Z{f3, X) the resultant /3 will depend on the observed data 
X. Finally, the new marginal distribution for 9 is 



(6.102) 



For /? = (no moment constraint) we recover Bayes' rule. For /? 7^ Bayes' 
rule is modified by a "canonical" exponential factor. 



6.6 Commuting and non- commuting constraints 

The ME method allows one to process information in the form of constraints. 
When we are confronted with several constraints we must be particularly cau- 
tious. In what order should they be processed? Or should they be processed 
together? The answer depends on the problem at hand. (Here we follow [Giffin 
Caticha 07].) 

We refer to constraints as commuting when it makes no difference whether 
they are handled simultaneously or sequentially. The most common example 
is that of Bayesian updating on the basis of data collected in multiple experi- 
ments: for the purpose of inferring 9 it is well-known that the order in which 
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{x'i,x'2, . . .} is processed does not matter. (See section 
2.10.3.) The proof that ME is completely compatible with Bayes' rule implies 
that data constraints implemented through S functions, as in ( 6.80[ ), commute. 
It is useful to see how this comes about. 

When an experiment is repeated it is common to refer to the value of x in 
the first experiment and the value of x in the second experiment. This is a dan- 
gerous practice because it obscures the fact that we are actually talking about 
two separate variables. We do not deal with a single x but with a composite 
x = {xi,X2) and the relevant space is Xi x X2 x Q. After the first experiment 
yields the value Xi, represented by the constraint Ci : P{xi) = 6{xi — Xi), we 
can perform a second experiment that yields X2 and is represented by a sec- 
ond constraint C2 : P{x2) = 5{x2 — ^2)- These constraints ci and C2 commute 
because they refer to different variables xi and X2- An experiment, once per- 
formed and its outcome observed, cannot be un-performed and its result cannot 
be un-observed by a second experiment. Thus, imposing the second constraint 
does not imply a revision of the first. 

In general constraints need not commute and when this is the case the order 
in which they are processed is critical. For example, suppose the prior is q and 
we receive information in the form of a constraint, Ci. To update we maximize 
the entropy S[p,q] subject to Ci leading to the posterior Pi as shown in Figure 
6.1. Next we receive a second piece of information described by the constraint 
C2. At this point we can proceed in essentially two different ways: 

(a) Sequential updating. Having processed Ci, we use Pi as the current 
prior and maximize S[p, Pi] subject to the new constraint C2. This leads us to 
the posterior P^. 

(b) Simultaneous updating. Use the original prior q and maximize S[p, q] 
subject to both constraints Ci and C2 simultaneously. This leads to the poste- 
rior Ph0 

To decide which path (a) or (b) is appropriate we must be clear about how 
the ME method handles constraints. The ME machinery interprets a constraint 
such as Ci in a very mechanical way: all distributions satisfying Ci are in 
principle allowed and all distributions violating Ci are ruled out. Updating to 
a posterior Pi consists precisely in revising those aspects of the prior q that 
disagree with the new constraint Ci. However, there is nothing final about the 
distribution Pi . It is just the best we can do in our current state of knowledge 
and it may happen that future information will require us to revise it further. 
Indeed, when new information C2 is received we must reconsider whether the 
original constraint Ci remains valid or not. Are all distributions satisfying the 
new C2 really allowed, even those that violate Ci? If we decide that this is the 
case then the new C2 takes over and we update from the current Pi to the final 
posterior Pq. The old constraint Ci may still exert some limited infiuence on 



■^At first sight it migiit appear that there exists a third possibility of simultaneous updating: 
(c) use Pi as the current prior and maximize S[p, Pi] subject to both constraints Ci and C2 
simultaneously. Fortunately, and this is a valuable check for the consistency of the ME method, 
it is easy to show that case (c) is equivalent to case (b). Whether we update from q or from 
Pi the selected posterior is Pi,. 




Figure 6.1: Illustrating the difference between processing two constraints Ci and 
C2 sequentially {q ^ Pi ^ Pa) and simultaneously {q ^ Pb or q ^ Pi ^ Pt). 



the final posterior Pa through its effect on the intermediate posterior Pi, but 
from now on C'l is considered obsolete and will be ignored. 

Alternatively, we may decide that the old constraint Ci retains its validity. 
The new C2 is not meant to replace the old Ci but to provide an additional 
refinement of the family of allowed posteriors. If this is the case, then the 
constraint that correctly reflects the new information is not C2 but the more 
restrictive Ci A C2. The two constraints Ci and C2 should be processed simul- 
taneously to arrive at the correct posterior Pf,. 

To summarize: sequential updating is appropriate when old constraints be- 
come obsolete and are superseded by new information; simultaneous updating is 
appropriate when old constraints remain valid. The two cases refer to different 
states of information and therefore we expect that they will result in different 
inferences. These comments are meant to underscore the importance of under- 
standing what information is being processed; failure to do so will lead to errors 
that do not reflect a shortcoming of the ME method but rather a misapplication 
of it. 



6.7 Information geometry 

This section provides a very brief introduction to an important subject that 
deserves a much more extensive treatment. [Amari 85, Amari Nagaoka 00] 

Consider a family of distributions p{x\9) labelled by a finite number of pa- 
rameters 0*, i = 1 . . . rt. It is usually possible to think of the family of distribu- 



144 



CHAPTER 6. ENTROPY III: UPDATING PROBABILITIES 



tions p(x\9) as a manifold - an n-dimensional space that is locally isomorphic 
to IR"pl The distributions p{x\9) are points in this "statistical manifold" with 
coordinates given by the parameters 9^ . We can introduce the idea of a distance 
between two such points - that is, a 'distance' between probability distribu- 
tions. The distance d£ between two neighboring points 9 and 9 + d9 is given by 
a generalization of Pythagoras' theorem in terms of a metric tensor g^j 

df = gijd&'dO^ . (6.103) 

The singular importance of the metric tensor gij derives from a most remarkable 
theorem due to Cencov that we mention without proof. [Cencov 81, Campbell 
86] The theorem states that the metric gij on the manifold of probability distri- 
butions is unique: there is only one metric that takes into account the fact that 
these are not distances between simple structureless dots but between probabil- 
ity distributions. Up to a scale factor, which merely reflects a choice of units, 
the unique distance is given by the information metric which we introduce below 
in three independent but intuitively appealing ways. 



6.7.1 Derivation from distinguishability 

We seek a quantitative measure of the extent that two distributions p{x\9) and 
p{x\9+d9) can be distinguished. The following argument is intuitively appealing. 
Consider the relative difference, 

p{x\9) 89' ^ ' 

The expected value of the relative difference might seem a good candidate, but 
it does not work because it vanishes identically, 

dxp{x\9) dO^ ^'^^^wj '^''^^^'^^ " ^^-^^^^ 

However, the variance does not vanish, 

dxp{x\9) d\ogp{x\9) 

' ' 89' 89^ ^ ' 

This is the measure of distinguishability we seek; a small value of d(^ means 
the points 9 and 9 + dO are difficult to distinguish. It suggests introducing the 
matrix g^j 

[a ( \a\^^^^v{A^) d^o%p{x\9) /R1nv^ 
= y dxp{x\B) (6.107) 

^Of course it is possible to conceive of sufficiently singular families of distributions that are 
not smooth manifolds. This does not detract from the value of the methods of information 
geometry any more than the existence of spaces with complicated geometries detracts from 
the general value of geometry itself. 

^The use of superscripts rather than subscripts for the indices labelling coordinates is 
a standard and very convenient notational convention in differential geometry. We adopt 
the standard Einstein convention of summing over repeated indices when one appears as a 
superscript and the other as a subscript. 
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called the Fisher information matrix [Fisher 25] , so that 

df ^g.jdO'de^ . (6.108) 

Up to now no notion of distance has been introduced. Normally one says that 
the reason it is difficult to distinguish two points in say, the three dimensional 
space we seem to inhabit, is that they happen to be too close together. It is very 
tempting to invert the logic and assert that the two points 9 and 9 + d9 must 
be very close together because they are difficult to distinguish. Furthermore, 
note that being a variance d£'^ is positive and vanishes only when d0 vanishes. 
Thus it is natural to interpret gij as the metric tensor of a Riemannian space 
[Rao 45]. It is known as the information metric. The recognition by Rao that 
gij is a metric in the space of probability distributions gave rise to the sub- 
ject of information geometry [Amari 85] , namely, the application of geometrical 
methods to problems in inference and in information theory. A disadvantage 
of this heuristic argument is that it does not make explicit a crucial property 
mentioned above that except for an overall multiplicative constant this metric 
is unique. [Cencov 81, Campbell 86] 

The coordinates 9 are quite arbitrary; one can freely relabel the points in the 
manifold. It is then easy to check that gij are the components of a tensor and 
that the distance di'^ is an invariant, a scalar under coordinate transformations. 



6.7.2 Derivation from a Euclidean metric 

Consider a discrete variable a = 1 . . .n. The restriction to discrete variables 
is not a serious limitation, we can choose n sufficiently large to approximate a 
continuous distribution to any desired degree. The possible probability distri- 
butions of a can be labelled by the probability values themselves: a probability 
distribution can be specified by a point p with coordinates {p^ . . .p"). The cor- 
responding statistical manifold is the simplex 5„_i = {p = {p^ . . . p") : J2 v"" — 
!}• 

1/2 

Next we change to new coordinates i/''' — (p") . In these new coordi- 
nates the equation for the simplex Sn-\ - the normalization condition - reads 

(V'°) = 1, which we recognize as the equation of an {n — l)-sphere embed- 
ded in an n-dimensional Euclidean space M", provided the ■0" are interpreted 
as Cartesian coordinates. This suggests that we assign the simplest possible 
metric: the distance between the distribution p{^) and its neighbor p('0 -I- dip) 
is the Euclidean distance in M", 

df = J2 {di'^f = Sabdip^dip^ . (6.109) 

a 

Distances between more distant distributions are merely angles defined on the 
surface of the sphere 



Except for an overall constant this is the same information metric (6.1081 we 
defined earlier! Indeed, consider an m-dimensional subspace (m < n — V) oi the 
sphere Sn-i defined by -0 = ''/'(^^i • • ■ ; 9™). The parameters 9^ , i — I . . . m, can 
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be used as coordinates on the subspace. The Euclidean metric on induces a 
metric on the subspace. The distance between p{9) and p{9 + dO) is 

df = Sabdrd^' = Sab^de^^de^ 



which (except for the factor 1/4) we recognize as the discrete version of (6.1071 



and (6.1081. This interesting result does not constitute a "derivation." There is 
a priori no reason why the coordinates -0 should be singled out as special and 
attributed a Euclidean metric. But perhaps it helps to lift the veil of mystery 



that might otherwise surround the strange expression (6.1071 



6.7.3 Derivation from relative entropy 



The "derivation" that follows has the merit of drawing upon our intuition about 
relative entropy. Consider the entropy of one distribution p{x\9') relative to 
another p{x\6), 

p{x\B') 



S{9\0) = ~ j dxp{x\e')\og 



p{x\9) 



(6.111) 



We study how this entropy varies when 6' = 9 + dO is in the close vicinity of a 
given 9. As we had seen in section 4.2 - recall the Gibbs inequality S{9' , 6) <Q 
with equality if and only \i 9' = 9 - the entropy S{6\ 9) attains an absolute 
maximum aX 9' = 6 . Therefore, the first nonvanishing term in the Taylor 
expansion about 9 is second order in d9 



S{9 + d9, 9) = 



ldS{9\9) 



d9'd9^ 



< 



2 d9'W9'^ 

and we use this quadratic form to define the information metric, 



dcf 03(9', 9) 

9ij = 



so that 



d9'W9'3 



S{9 + d9,9) = --df 



(6.112) 

(6.113) 
(6.114) 



It is straightforward to show that (6.1131 coincides with (6.1071. 



6.7.4 Volume elements in curved spaces 

Having decided on a measure of distance we can now also measure angles, areas, 
volumes and all sorts of other geometrical quantities. Here we only consider 
calculating the m-dimensional volume of the manifold of distributions p{x\9) 
labelled by parameters 9'' with i = I . . .m. 
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The parameters 6^ are coordinates for the point p and in these coordinates it 
may not be obvious how to write down an expression for a volume element dV. 
But within a sufficiently small region - which is what a volume element is - any 
curved space looks flat. Curved spaces are 'locally flat'. The idea then is rather 
simple; within that very small region we should use Cartesian coordinates and 
the metric takes a very simple form, it is the identity matrix, dab- In locally 
Cartesian coordinates (f)" the volume element is simply given by the product 

dV = d(j)^d(j)'^ ...dcj)"" , (6.115) 

which, in terms of the old coordinates 6^, is 

d'^e . (6.116) 

This is the volume we seek written in terms of the coordinates 9. Our remaining 
problem consists in calculating the Jacobian \d(p/d9\ of the transformation that 
takes the metric gij into its Euclidean form Sab- 
Let the locally Cartesian coordinates be defined by 4>°- = $°(6'^, . . .9"^). A 
small change in d9 corresponds to a small change in d<p, 

dr = Xtd9^ where = ^ , (6.117) 
and the Jacobian is given by the determinant of the matrix Xf, 

= \det{Xt)\ . (6.118) 

The distance between two neighboring points is the same whether we compute 
it in terms of the old or the new coordinates, 

de = gijd9'd9^ = 6abd<f)''d(f)'' (6.119) 

Thus the relation between the old and the new metric is 

9ij = SabXl'X^ . (6.120) 

The right hand side represents the product of three matrices. Taking the deter- 
minant we get 

5 =' det(5a6) = [det (Xf )]' , (6.121) 

so that 

|det(X,")|=5^/2. (6.122) 

We have succeeded in expressing the volume element totally in terms of the 
coordinates 9 and the known metric gij{9). The answer is 



dV 



89 



d9^d9'^ ...d9" 



d(t> 
89 




dV = g^l^{9)d"'9 



(6.123) 
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The volume of any extended region on the manifold is 

V = j dV = j g^/\0)d"e . (6.124) 

After this technical detour we are now ready to return to the main subject of 
this chapter - updating probabilities - and derive one last and very important 
feature of the ME method. 



6.8 Deviations from maximum entropy 

There is one last issue that must be addressed before one can claim that the 
design of the ME method is more or less complete. Higher entropy represents 
higher preference but there is nothing in the previous arguments to tell us by 
how much. Does twice the entropy represent twice the preference or four times 
as much? We can rank probability distributions p relative to a prior q according 
to the relative entropy S\p, q] but any monotonic function of the relative entropy 
will accomplish the same goal. Once we have decided that the distribution of 
maximum entropy is to be preferred over all others the following question arises: 
Suppose the maximum of the entropy function is not particularly sharp, are we 
really confident that distributions with entropy close to the maximum are totally 
ruled out? Can we quantify 'preference' ? We want a quantitative measure of the 
extent to which distributions with lower entropy are ruled out. The discussion 
below follows [Caticha 00]. 

Suppose we have maximized the entropy S\p, q] subject to certain constraints 
and obtain a probability distribution po{x). The question we now address con- 
cerns the extent to which po{x) should be preferred over other distributions 
with lower entropy. Consider a family of distributions p{x\9) labelled by a finite 
number of parameters 9' , i = 1 . . .n. We assume that the p{x\9) satisfy the 
same constraints that led us to select po{x) and that po{x) itself is included in 
the family. Further we choose the parameters 6 so that po{x) = p{x\9 = 0). The 
question about the extent that p{x\9 = 0) is to be preferred over p{x\9 / 0) is a 
question about the probability p{9) of various values of 9: what is the rational 
degree of behef that the selected value should be 9? The original problem which 
led us to design the niaxiniuni entropy method was to assign a probability to x; 
we now see that the full problem is to assign probabilities to both x and 9. We 
are concerned not just with p{x) but rather with the joint distribution P{x, 9); 
the universe of discourse has been expanded from X (the space of a;s) to x 6 
(8 is the space of parameters 9) . 

To determine the joint distribution P{x, 9) we make use of essentially the 
only method at our disposal - the ME method itself - but this requires that 
we address the standard two preliminary questions: First, what is the prior 
distribution, what do we know about x and before we receive information 
about the constraints? And second, what is this new information that constrains 
the allowed P{x,9)7 
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This first question is the more subtle one: when we know absohitely nothing 
about the 9s we know neither their physical meaning nor whether there is any 
relation to the a;s. A prior that reflects this lack of correlations is a product, 
q{x,0) = q{x)^i{6). Wc will assume that the prior over x is known - it is the 
prior we had used when we updated from q{x) to Pq{x). Since we are totally 
ignorant about 6 we would like to choose /u(^) so that it reflects a uniform 
distribution but here wc stumble upon a problem: uniform means that equal 
volumes in Q are assigned equal probabilities and knowing nothing about the 
9s we do not yet know what "equal" volumes in 6 could possibly mean. We 
need some additional information. 

Suppose next that we are told that the 9s represent probability distributions, 
they are parameters labeling some unspecifled distributions p{x\9). We do not 
yet know the functional form of p{x\9), but if the 9s derive their meaning solely 
from the p{x\9) then there exists a natural measure of distance in the space 
6. It is the information metric gij{6) introduced in the previous section and 
the corresponding volume elements are given by g^/'^{9)cr^9, where g{9) is the 
determinant of the metric. The uniform prior for 9, which assigns equal prob- 
abilities to equal volumes, is proportional to g^/^{9) and therefore we choose 

Next we tackle the second question: what are the constraints on the allowed 
joint distributions p{x\6)? Consider the space of all joint distributions. To 
each choice of the functional form of p{x\9) (whether we talk about Gaussians, 
Boltzmann-Gibbs distributions, or something else) there corresponds a different 
subspace deflned by distributions of the form P{x, 9) = p{9)p{x\9). The crucial 
constraint is that which specifies the subspace, that is, the particular functional 
form for p{x\9). This defines the meaning to the 9s - for example, the 9s could 
be the mean and variance in Gaussian distributions, or Lagrange multipliers 
in Boltzmann-Gibbs distributions. It also fixes the prior /i(6') on the relevant 
subspace. Notice that the kind of constraint that we impose here is very different 
from those that appear in usual applications of maximum entropy method, which 
are in the form of expectation values. 

To select the preferred distribution P{x, 9) we maximize the entropy S[P\g^^'^q\ 
over all distributions of the form P{x,9) = p{6)p{x\6) by varying with respect 
to p{6) with p{x\9) flxed. It is convenient to write the entropy as 

S[P,9'^'l] = -j dxd9p{9)p{x\9) log^^^^ 

= S\p,g'/^] + Jd9p{9)S{e), (6.125) 

where 

S\p,g'/'] = -j d9p{e)\og^^^ (6.126) 

and 

S(9) = - j dxp{x\9)\og^'^^. (6.127) 
J q{x) 
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The notation shows that S[p, g^l'^\ is a functional oip{9) while S{9) is a function 



of 9 (it is also a functional of p{x\9)). Maximizing (6.1251 with respect to 
variations Sp{9) such that J d9p{9) = 1, yields 

= J d9 (^-log-^ + 3(9) + log Sp{9), (6.128) 

where the required Lagrange multiplier has been written as 1 — log (. Therefore 
the probability that the value of 9 should lie within the small volume g^/^{9)d^9 
is 

p(6))d"6l= ^ e^(^)(7i/2(5,)^«g, ^j^j^ j creg^/'^{9)e^'^'^\ (6.129) 



Equation (6.129 1 is the result we seek. It tells us that, as expected, the preferred 



value of 9 is that which maximizes the entropy S{9)^ eq.(6.127l, because this 
maximizes the scalar probability density exp S{9). But it also tells us the degree 
to which values of 9 away from the maximum are ruled out. For macroscopic 



systems the preference for the ME distribution can be overwhelming. Eq.(6.129 1 
agrees with the Einstein thermodynamic fluctuation theory and extends it be- 
yond the regime of small fluctuations - in the next section we deal with fluctua- 
tions as an illustration. Note also that the density exp S(9) is a sca lar function 



and the presence of the Jacobian factor g^^^{9) makes Eq.(6.129l manifestly 
invariant under changes of the coordinates 9^ in the space Q. 

We conclude this section by pointing out that there are a couple of interesting 
points of analogy between the pair {maximum likelihood method/Bayes' rule} 
on one hand and the corresponding pair {MaxEnt/ME} methods on the other 

hand. Note that maximizing the likelihood function L{9\x) p{x\9) selects a 
single preferred value of 9 but no measure is given of the extent to which other 
values of 9 are ruled out. The method of maximum likelihood does not provide 
us with a distribution for 9 ~ the likelihood function L{9\x) is not a probability 
distribution for 9. Similarly, maximizing entropy as prescribed by the MaxEnt 
method yields a single preferred value of the label 9 but MaxEnt fails to address 
the question of the extent to which other values of 9 are ruled out. Neither 
Bayes' rule nor the ME method suffer from this limitation. 

The second point of analogy is that neither the maximum likelihood nor 
the MaxEnt methods are capable of handling information contained in prior 
distributions, while both Bayesian and ME methods can. This limitation of 
maximum likelihood and MaxEnt is not surprising since neither method was 
designed for updating probabilities. 



6.9 An application to fluctuations 

The starting point for the standard formulation of the theory of fluctuations in 
thermodynamic systems (see [Landau 77, Callen 85]) is Einstein's inversion of 
Boltzmann's formula S = k log W to obtain the probability of a fluctuation in 
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the form W ~ expS/k. A careful justification, however, reveals a number of 
approximations which, for most purposes, are legitimate and work very well. A 
re-examination of fluctuation theory from the point of view of ME is, however, 
valuable. Our general conclusion is that the ME point of view allows exact 
formulations; in fact, it is clear that deviations from the canonical predictions 
can be expected, although in general they will be negligible. Other advantages of 
the ME approach include the explicit covariance under changes of coordinates, 
the absence of restrictions to the vicinity of equilibrium or to large systems, and 
the conceptual ease with which one deals with fluctuations of both the extensive 
as well as their conjugate intensive variables. [Caticha 00] 

This last point is an important one: within the canonical formalism (section 
4.8) the extensive variables such as energy are uncertain while the intensive ones 
such as the temperature or the Lagrange multiplier P are fixed parameters, they 
do not fluctuate. There are, however, several contexts in which it makes sense 
to talk about fluctuations of the conjugate variables. We discuss the standard 
scenario of an open system that can exchange say, energy, with its environment. 

Consider the usual setting of a thermodynamical system with microstates 
labelled by x. Let m{x)dx be the number of microstates within the range dx. 
According to the postulate of "equal a priori probabilities" we choose a uniform 
prior distribution proportional to the density of states m{x). The canonical 
ME distribution obtained by maximizing S[p, m] subject to constraints on the 
expected values (/'") = of relevant variables f''{x), is 

p(x\F) ^ ^—m{x)e-^''f''^''^ with Z{X) = Jdxm{x) e-^''^"^'"'' , (6.130) 

and the corresponding entropy is 

S{F) = log Z{X) + XkF'' . (6.131) 

Fluctuations of the variables f^{x) or of any other function of the microstate 
X are usually computed in terms of the various moments of p{x\F). Within this 
context all expected values such as the constraints {f^) = F^ and the entropy 
S{F) itself are fixed; they do not fluctuate. The corres ponding conjugate vari- 



ables, the Lagrange multipliers Afc — dS/dF^ , eq.(4.77l, do not fluctuate either. 

The standard way to make sense of A fluctuations is to couple the system of 
interest to a second system, a bath, and allow exchanges of the quantities f*'. All 
quantities referring to the bath will be denoted by primes: the microstates are 
x', the density of states is m'{x'), and the variables are f'^{x'), etc. Even though 
the overall expected value {f^ + f'^) = F!^ of the combined system plus bath is 
fixed, the individual expected values (/'') = F'^ and (/"") = F"' = F^-F'^ are 
allowed to fluctuate. The ME distribution po{x,x') that best reflects the prior 
information contained in m(x) and m'(x') updated by information on the total 

T 

Po{x,x') = ^m(a;)m'(a;')e-^°°(^'(")+^"("')). (6.132) 



FM is 
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But less than ME distributions are not totally ruled out; to explore the possi- 
bility that the quantities are distributed between the two systems in a less 
than optimal way we consider distributions p{x, x' , F) constrained to the form 



P{x, x', F) = p{F)p{x\F)p{x'\Ft - F), 



(6.133) 



where p{x\F) is the canonical distribution in eq.(6.130l, its entropy is eq.(6.131 1 
and analogous expressions hold for the primed quantities. 

We are now ready to write down the probability that the value of F fluctuates 
into a small volume g^^'^{F)dF. From eq.(6.129l we have 



p{F)dF^ - e'^^^^'>g^/^{F)dF, 



(6.134) 



where is a normalization constant and the entropy St{F) of the system plus 
the bath is 

St{F) S{F) + S'{Ft - F). (6.135) 

The formalism simplifles considerably when the bath is large enough that ex- 
changes of F do not affect it, and A' remains fixed at Aq. Then 



S'{Ft - F) 



log Z'iXo) + Aofc (F*; - F'^) = const -AofcF^ 



(6.136) 



It remains to calculate the determinant g{F) of the information metric given 
by eq.doTsl, 



9^J 



d^STiF,F) 



S{F,F) + S'{Ft- F,Ft- F) (6.137) 



dF^dF^i dF^dFi 

where the dot indicates that the derivatives act on the first argument. The first 
term on the right is 



d^S{F,F) 



dF'dPi 
d^S{F) 



dxp{x\F) lo; 



p{x\F) m{x) 
m{x) p{x\F) 



dF^dFJ 

To calculate the integral on the right use 

p{x\F) 



^x':±'giog^(-'^) 



dFWFJ 



m{x) 



log 



m{x) 



logZiX)-Xkr{x) 



(6.138) 



(6.139) 



(from eq.(6.130l so that the integral vanishes, 



-logZ(A) 

Similarly 
92 



dF^dFi 



dF^dFi 



S'{Ft 



dxp{x\F) — Aj 



OF'dFJ 



dxp{x\F)f''{x) = . (6.140) 



F,Ft 



F) 



d^S'jFT - F) 
dF^dFJ 

d.'^!^m^log^(^'l^- 



(6.141) 

F) 



dFWFi 



m'(x') 
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and here, using eq.(6.136l, both terms vanish. Therefore 



1^- (6.142) 

We conclude that the probabihty that the value of F fluctuates into a small 
volume g^/'^{F)dF becomes 

p{F)dF = ^ e^'^^^-^o^^" g^'^{F)dF . (6.143) 

This equation is exact. 

An important difference with the usual theory stems from the presence of 
the Jacobian factor g^/^{F). This is required by coordinate invariance and can 
lead to small deviations from the canonical predictions. The quantities (Afe) and 
(F*"') may be close but will not in general coincide with the quantities Xok and Fq 
at the point where the scalar probability density attains its maximum. For most 
thermodynamic systems however the maximum is very sharp. In its vicinity the 
Jacobian can be considered constant, and one obtains the usual results [Landau 
77] , namely, that the probability distribution for the fluctuations is given by the 
exponential of a Legendre transform of the entropy. 

The remaining difficulties are purely computational and of the kind that 
can in general be tackled systematically using the method of steepest descent 
to evaluate the appropriate generating function. Since we are not interested 



in variables referring to the bath we can integrate Eq.(6.133l over x' , and use 
the distribution P(x,F) — p{F)p{x\F) to compute various moments. As an 
example, the correlation between 6Xi = Xi — (A^) and 5f^ = — (p) or 
SF^ = Fi - (Fi) is 

(SKSP) = (SKSF^) = + (Ao, - (A,)) (fJ - (F^)) . (6.144) 

When the differences Xq^ - {X.,) or F^ ~ (F^) are negligible one obtains the usual 
expression, 

(SX^SP) « -Si . (6.145) 



6.10 Conclusion 

Any Bayesian account of the notion of information cannot ignore the fact that 
Bayesians are concerned with the beliefs of rational agents. The relation be- 
tween information and beliefs must be clearly spelled out. The definition we 
have proposed - that information is that which constrains rational beliefs and 
therefore forces the agent to change its mind - is convenient for two reasons. 
First, the information/belief relation very explicit, and second, the definition is 
ideally suited for quantitative manipulation using the ME method. 

Dealing with uncertainty requires that one solve two problems. First, one 
must represent a state of knowledge as a consistent web of interconnected be- 
liefs. The instrument to do it is probability. Second, when new information 
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becomes available the beliefs must be updated. The instrument for this is rela- 
tive entropy. It is the only candidate for an updating method that is of universal 
applicability and obeys the moral injunction that one should not change one's 
mind frivolously. Prior information is valuable and should not be revised except 
when demanded by new evidence, in which case the revision is no longer optional 
but obligatory. The resulting general method - the ME method - can handle 
arbitrary priors and arbitrary constraints; it includes MaxEnt and Bayes' rule 
as special cases; and it provides its own criterion to assess the extent that non 
maximum-entropy distributions are ruled outj^ 

To conclude I cannot help but to express my continued sense of wonder and 
astonishment that the method for reasoning under uncertainty - which should 
presumably apply to the whole of science - turns out to rest upon an ethical 
foundation of intellectual honesty. The moral imperative is to uphold those 
beliefs and only those beliefs that obey very strict constraints of consistency; 
the allowed beliefs must be consistent among themselves, and they must be 
consistent with the available information. Just imagine the implications! 



^For possible developments and applications of these ideas, which we hope will be the 
subject of future additions to these lectures, see the "Suggestions for further reading." 
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